There are 1411 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
astropy__astropy-11693, astropy__astropy-12057, astropy__astropy-12318, astropy__astropy-12544, astropy__astropy-12825, astropy__astropy-12842, astropy__astropy-12880, astropy__astropy-12891, astropy__astropy-12907, astropy__astropy-12962, astropy__astropy-13033, astropy__astropy-13068, astropy__astropy-13073, astropy__astropy-13075, astropy__astropy-13132, astropy__astropy-13158, astropy__astropy-13162, astropy__astropy-13234, astropy__astropy-13236, astropy__astropy-13306, astropy__astropy-13398, astropy__astropy-13417, astropy__astropy-13438, astropy__astropy-13453, astropy__astropy-13462, astropy__astropy-13465, astropy__astropy-13469, astropy__astropy-13477, astropy__astropy-13638, astropy__astropy-13731, astropy__astropy-13734, astropy__astropy-13842, astropy__astropy-13933, astropy__astropy-13977, astropy__astropy-14042, astropy__astropy-14163, astropy__astropy-14182, astropy__astropy-14213, astropy__astropy-14253, astropy__astropy-14295, astropy__astropy-14365, astropy__astropy-14369, astropy__astropy-14379, astropy__astropy-14413, astropy__astropy-14439, astropy__astropy-14528, astropy__astropy-14566, astropy__astropy-14578, astropy__astropy-14590, astropy__astropy-14598, astropy__astropy-14701, astropy__astropy-14907, astropy__astropy-14966, astropy__astropy-7008, astropy__astropy-7441, astropy__astropy-7737, astropy__astropy-7746, astropy__astropy-7858, astropy__astropy-7973, astropy__astropy-8251, astropy__astropy-8519, astropy__astropy-8715, astropy__astropy-8747, django__django-10287, django__django-10531, django__django-10554, django__django-10643, django__django-10680, django__django-10737, django__django-10853, django__django-10904, django__django-10910, django__django-10939, django__django-10957, django__django-10999, django__django-11003, django__django-11019, django__django-11030, django__django-11057, django__django-11062, django__django-11070, django__django-11087, django__django-11088, django__django-11115, django__django-11129, django__django-11138, django__django-11169, django__django-11177, django__django-11205, django__django-11214, django__django-11239, django__django-11260, django__django-11265, django__django-11270, django__django-11278, django__django-11279, django__django-11281, django__django-11283, django__django-11299, django__django-11323, django__django-11354, django__django-11356, django__django-11359, django__django-11383, django__django-11396, django__django-11400, django__django-11417, django__django-11422, django__django-11423, django__django-11433, django__django-11446, django__django-11457, django__django-11477, django__django-11501, django__django-11517, django__django-11525, django__django-11527, django__django-11539, django__django-11543, django__django-11550, django__django-11559, django__django-11560, django__django-11564, django__django-11584, django__django-11591, django__django-11605, django__django-11622, django__django-11630, django__django-11638, django__django-11669, django__django-11677, django__django-11688, django__django-11692, django__django-11701, django__django-11728, django__django-11734, django__django-11740, django__django-11742, django__django-11751, django__django-11754, django__django-11772, django__django-11797, django__django-11808, django__django-11810, django__django-11820, django__django-11829, django__django-11885, django__django-11891, django__django-11893, django__django-11894, django__django-11905, django__django-11910, django__django-11911, django__django-11916, django__django-11983, django__django-11991, django__django-12009, django__django-12062, django__django-12073, django__django-12113, django__django-12122, django__django-12148, django__django-12153, django__django-12185, django__django-12187, django__django-12196, django__django-12212, django__django-12225, django__django-12262, django__django-12273, django__django-12281, django__django-12299, django__django-12313, django__django-12343, django__django-12360, django__django-12396, django__django-12406, django__django-12407, django__django-12431, django__django-12441, django__django-12464, django__django-12469, django__django-12470, django__django-12477, django__django-12484, django__django-12485, django__django-12504, django__django-12508, django__django-12513, django__django-12518, django__django-12519, django__django-12532, django__django-12553, django__django-12556, django__django-12588, django__django-12589, django__django-12591, django__django-12630, django__django-12663, django__django-12669, django__django-12733, django__django-12734, django__django-12747, django__django-12748, django__django-12754, django__django-12771, django__django-12774, django__django-12796, django__django-12821, django__django-12830, django__django-12851, django__django-12856, django__django-12869, django__django-12906, django__django-12908, django__django-12910, django__django-12912, django__django-12928, django__django-12936, django__django-12951, django__django-12953, django__django-12957, django__django-12961, django__django-12965, django__django-12973, django__django-13023, django__django-13030, django__django-13033, django__django-13066, django__django-13077, django__django-13097, django__django-13111, django__django-13112, django__django-13115, django__django-13118, django__django-13145, django__django-13162, django__django-13170, django__django-13192, django__django-13195, django__django-13199, django__django-13207, django__django-13212, django__django-13220, django__django-13233, django__django-13236, django__django-13237, django__django-13250, django__django-13265, django__django-13267, django__django-13287, django__django-13295, django__django-13297, django__django-13300, django__django-13321, django__django-13344, django__django-13346, django__django-13350, django__django-13355, django__django-13371, django__django-13406, django__django-13413, django__django-13426, django__django-13431, django__django-13448, django__django-13449, django__django-13454, django__django-13458, django__django-13460, django__django-13466, django__django-13484, django__django-13490, django__django-13495, django__django-13512, django__django-13513, django__django-13528, django__django-13530, django__django-13551, django__django-13560, django__django-13568, django__django-13578, django__django-13585, django__django-13589, django__django-13592, django__django-13606, django__django-13607, django__django-13660, django__django-13671, django__django-13682, django__django-13684, django__django-13708, django__django-13714, django__django-13722, django__django-13744, django__django-13768, django__django-13791, django__django-13794, django__django-13797, django__django-13800, django__django-13808, django__django-13886, django__django-13915, django__django-13924, django__django-13992, django__django-14011, django__django-14019, django__django-14026, django__django-14030, django__django-14031, django__django-14034, django__django-14056, django__django-14059, django__django-14071, django__django-14109, django__django-14122, django__django-14124, django__django-14149, django__django-14155, django__django-14164, django__django-14170, django__django-14182, django__django-14241, django__django-14271, django__django-14282, django__django-14311, django__django-14313, django__django-14315, django__django-14324, django__django-14336, django__django-14372, django__django-14374, django__django-14376, django__django-14385, django__django-14387, django__django-14395, django__django-14396, django__django-14399, django__django-14404, django__django-14407, django__django-14430, django__django-14434, django__django-14453, django__django-14463, django__django-14471, django__django-14480, django__django-14495, django__django-14508, django__django-14513, django__django-14518, django__django-14534, django__django-14580, django__django-14631, django__django-14634, django__django-14664, django__django-14667, django__django-14681, django__django-14717, django__django-14722, django__django-14725, django__django-14727, django__django-14730, django__django-14751, django__django-14762, django__django-14785, django__django-14792, django__django-14802, django__django-14805, django__django-14832, django__django-14861, django__django-14871, django__django-14878, django__django-14880, django__django-14890, django__django-14894, django__django-14916, django__django-14919, django__django-14935, django__django-14969, django__django-14983, django__django-14996, django__django-14997, django__django-15018, django__django-15031, django__django-15037, django__django-15038, django__django-15061, django__django-15087, django__django-15098, django__django-15108, django__django-15127, django__django-15135, django__django-15136, django__django-15139, django__django-15154, django__django-15166, django__django-15180, django__django-15199, django__django-15202, django__django-15240, django__django-15248, django__django-15252, django__django-15268, django__django-15272, django__django-15280, django__django-15316, django__django-15318, django__django-15320, django__django-15324, django__django-15334, django__django-15352, django__django-15375, django__django-15388, django__django-15401, django__django-15413, django__django-15421, django__django-15423, django__django-15433, django__django-15438, django__django-15481, django__django-15483, django__django-15492, django__django-15503, django__django-15554, django__django-15563, django__django-15576, django__django-15613, django__django-15620, django__django-15629, django__django-15648, django__django-15651, django__django-15666, django__django-15669, django__django-15671, django__django-15678, django__django-15682, django__django-15689, django__django-15695, django__django-15703, django__django-15732, django__django-15738, django__django-15747, django__django-15752, django__django-15766, django__django-15774, django__django-15781, django__django-15819, django__django-15869, django__django-15916, django__django-15957, django__django-15969, django__django-15973, django__django-15993, django__django-15995, django__django-15996, django__django-16027, django__django-16032, django__django-16037, django__django-16072, django__django-16076, django__django-16082, django__django-16092, django__django-16117, django__django-16120, django__django-16142, django__django-16143, django__django-16208, django__django-16229, django__django-16256, django__django-16263, django__django-16281, django__django-16302, django__django-16311, django__django-16322, django__django-16343, django__django-16369, django__django-16398, django__django-16400, django__django-16408, django__django-16411, django__django-16501, django__django-16502, django__django-16514, django__django-16517, django__django-16532, django__django-16560, django__django-16578, django__django-16597, django__django-16599, django__django-16603, django__django-16614, django__django-16629, django__django-16631, django__django-16635, django__django-16649, django__django-16667, django__django-16686, django__django-16707, django__django-16735, django__django-16745, django__django-16746, django__django-16757, django__django-16759, django__django-16786, django__django-16810, django__django-16816, django__django-16820, django__django-16830, django__django-16865, django__django-16879, django__django-16883, django__django-16903, django__django-16910, django__django-16920, django__django-16938, django__django-16948, django__django-16950, django__django-16952, django__django-16983, django__django-17045, django__django-17046, django__django-17058, django__django-17066, django__django-5158, django__django-5470, django__django-7188, django__django-8119, django__django-8326, django__django-8630, django__django-9703, matplotlib__matplotlib-13859, matplotlib__matplotlib-13908, matplotlib__matplotlib-13913, matplotlib__matplotlib-13959, matplotlib__matplotlib-13980, matplotlib__matplotlib-13983, matplotlib__matplotlib-13984, matplotlib__matplotlib-14471, matplotlib__matplotlib-14623, matplotlib__matplotlib-17810, matplotlib__matplotlib-18869, matplotlib__matplotlib-19553, matplotlib__matplotlib-19743, matplotlib__matplotlib-19763, matplotlib__matplotlib-20374, matplotlib__matplotlib-20470, matplotlib__matplotlib-20518, matplotlib__matplotlib-20676, matplotlib__matplotlib-20679, matplotlib__matplotlib-20693, matplotlib__matplotlib-20761, matplotlib__matplotlib-20788, matplotlib__matplotlib-20816, matplotlib__matplotlib-21238, matplotlib__matplotlib-21318, matplotlib__matplotlib-21443, matplotlib__matplotlib-21490, matplotlib__matplotlib-21550, matplotlib__matplotlib-21559, matplotlib__matplotlib-21568, matplotlib__matplotlib-21570, matplotlib__matplotlib-21617, matplotlib__matplotlib-22711, matplotlib__matplotlib-22767, matplotlib__matplotlib-22815, matplotlib__matplotlib-22835, matplotlib__matplotlib-22871, matplotlib__matplotlib-22883, matplotlib__matplotlib-22929, matplotlib__matplotlib-22945, matplotlib__matplotlib-22991, matplotlib__matplotlib-23047, matplotlib__matplotlib-23057, matplotlib__matplotlib-23088, matplotlib__matplotlib-23140, matplotlib__matplotlib-23198, matplotlib__matplotlib-23266, matplotlib__matplotlib-23267, matplotlib__matplotlib-23288, matplotlib__matplotlib-23332, matplotlib__matplotlib-23348, matplotlib__matplotlib-23476, matplotlib__matplotlib-23516, matplotlib__matplotlib-23573, matplotlib__matplotlib-23740, matplotlib__matplotlib-23742, matplotlib__matplotlib-24013, matplotlib__matplotlib-24088, matplotlib__matplotlib-24111, matplotlib__matplotlib-24177, matplotlib__matplotlib-24224, matplotlib__matplotlib-24250, matplotlib__matplotlib-24257, matplotlib__matplotlib-24265, matplotlib__matplotlib-24538, matplotlib__matplotlib-24604, matplotlib__matplotlib-24619, matplotlib__matplotlib-24691, matplotlib__matplotlib-24749, matplotlib__matplotlib-24849, matplotlib__matplotlib-24870, matplotlib__matplotlib-24912, matplotlib__matplotlib-24924, matplotlib__matplotlib-24971, matplotlib__matplotlib-25027, matplotlib__matplotlib-25079, matplotlib__matplotlib-25126, matplotlib__matplotlib-25129, matplotlib__matplotlib-25238, matplotlib__matplotlib-25281, matplotlib__matplotlib-25311, matplotlib__matplotlib-25334, matplotlib__matplotlib-25346, matplotlib__matplotlib-25405, matplotlib__matplotlib-25430, matplotlib__matplotlib-25433, matplotlib__matplotlib-25479, matplotlib__matplotlib-25498, matplotlib__matplotlib-25515, matplotlib__matplotlib-25547, matplotlib__matplotlib-25551, matplotlib__matplotlib-25565, matplotlib__matplotlib-25624, matplotlib__matplotlib-25631, matplotlib__matplotlib-25640, matplotlib__matplotlib-25651, matplotlib__matplotlib-25712, matplotlib__matplotlib-25772, matplotlib__matplotlib-25779, matplotlib__matplotlib-25785, matplotlib__matplotlib-25794, matplotlib__matplotlib-25859, matplotlib__matplotlib-25960, matplotlib__matplotlib-26024, matplotlib__matplotlib-26089, matplotlib__matplotlib-26101, matplotlib__matplotlib-26122, matplotlib__matplotlib-26160, matplotlib__matplotlib-26184, matplotlib__matplotlib-26208, matplotlib__matplotlib-26249, matplotlib__matplotlib-26285, matplotlib__matplotlib-26341, matplotlib__matplotlib-26399, matplotlib__matplotlib-26466, matplotlib__matplotlib-26469, matplotlib__matplotlib-26472, matplotlib__matplotlib-26479, mwaskom__seaborn-2576, mwaskom__seaborn-2766, mwaskom__seaborn-2813, mwaskom__seaborn-2846, mwaskom__seaborn-2848, mwaskom__seaborn-2946, mwaskom__seaborn-2979, mwaskom__seaborn-3069, mwaskom__seaborn-3180, mwaskom__seaborn-3187, mwaskom__seaborn-3202, mwaskom__seaborn-3216, mwaskom__seaborn-3217, mwaskom__seaborn-3394, mwaskom__seaborn-3407, pallets__flask-4045, pallets__flask-4074, pallets__flask-4544, pallets__flask-4575, pallets__flask-4642, pallets__flask-4992, pallets__flask-5063, psf__requests-1339, psf__requests-1376, psf__requests-1657, psf__requests-1776, psf__requests-1944, psf__requests-2148, psf__requests-2466, psf__requests-2678, psf__requests-2754, psf__requests-2873, psf__requests-4718, psf__requests-6028, pydata__xarray-2922, pydata__xarray-3095, pydata__xarray-3114, pydata__xarray-3156, pydata__xarray-3159, pydata__xarray-3239, pydata__xarray-3302, pydata__xarray-3338, pydata__xarray-3364, pydata__xarray-3520, pydata__xarray-3527, pydata__xarray-3631, pydata__xarray-3637, pydata__xarray-3649, pydata__xarray-3733, pydata__xarray-3976, pydata__xarray-3979, pydata__xarray-3993, pydata__xarray-4094, pydata__xarray-4184, pydata__xarray-4248, pydata__xarray-4339, pydata__xarray-4419, pydata__xarray-4423, pydata__xarray-4442, pydata__xarray-4493, pydata__xarray-4510, pydata__xarray-4684, pydata__xarray-4695, pydata__xarray-4750, pydata__xarray-4758, pydata__xarray-4759, pydata__xarray-4767, pydata__xarray-4819, pydata__xarray-4827, pydata__xarray-4879, pydata__xarray-4911, pydata__xarray-4939, pydata__xarray-4940, pydata__xarray-5126, pydata__xarray-5187, pydata__xarray-5233, pydata__xarray-5362, pydata__xarray-5365, pydata__xarray-5455, pydata__xarray-5580, pydata__xarray-5662, pydata__xarray-6135, pydata__xarray-6386, pydata__xarray-6400, pydata__xarray-6548, pydata__xarray-6598, pydata__xarray-6721, pydata__xarray-6798, pydata__xarray-6804, pydata__xarray-6823, pydata__xarray-6857, pydata__xarray-6889, pydata__xarray-6938, pydata__xarray-6971, pydata__xarray-6992, pydata__xarray-6999, pydata__xarray-7003, pydata__xarray-7019, pydata__xarray-7052, pydata__xarray-7089, pydata__xarray-7101, pydata__xarray-7105, pydata__xarray-7112, pydata__xarray-7120, pydata__xarray-7147, pydata__xarray-7150, pydata__xarray-7179, pydata__xarray-7229, pydata__xarray-7347, pydata__xarray-7400, pydata__xarray-7444, pylint-dev__pylint-4175, pylint-dev__pylint-4330, pylint-dev__pylint-4339, pylint-dev__pylint-4398, pylint-dev__pylint-4421, pylint-dev__pylint-4492, pylint-dev__pylint-4516, pylint-dev__pylint-4551, pylint-dev__pylint-4604, pylint-dev__pylint-4661, pylint-dev__pylint-4669, pylint-dev__pylint-4703, pylint-dev__pylint-4858, pylint-dev__pylint-5175, pylint-dev__pylint-5201, pylint-dev__pylint-5231, pylint-dev__pylint-5446, pylint-dev__pylint-5613, pylint-dev__pylint-5730, pylint-dev__pylint-5743, pylint-dev__pylint-5839, pylint-dev__pylint-5951, pylint-dev__pylint-6059, pylint-dev__pylint-6196, pylint-dev__pylint-6357, pylint-dev__pylint-6358, pylint-dev__pylint-6412, pylint-dev__pylint-6506, pylint-dev__pylint-6517, pylint-dev__pylint-6526, pylint-dev__pylint-6556, pylint-dev__pylint-6820, pylint-dev__pylint-7097, pylint-dev__pylint-7228, pylint-dev__pylint-8124, pylint-dev__pylint-8169, pylint-dev__pylint-8683, pylint-dev__pylint-8757, pylint-dev__pylint-8799, pylint-dev__pylint-8819, pylint-dev__pylint-8898, pylint-dev__pylint-8929, pytest-dev__pytest-10115, pytest-dev__pytest-10343, pytest-dev__pytest-10356, pytest-dev__pytest-10371, pytest-dev__pytest-10442, pytest-dev__pytest-10482, pytest-dev__pytest-10758, pytest-dev__pytest-10893, pytest-dev__pytest-10988, pytest-dev__pytest-11041, pytest-dev__pytest-11044, pytest-dev__pytest-11047, pytest-dev__pytest-11125, pytest-dev__pytest-11148, pytest-dev__pytest-11160, pytest-dev__pytest-5103, pytest-dev__pytest-5205, pytest-dev__pytest-5221, pytest-dev__pytest-5254, pytest-dev__pytest-5281, pytest-dev__pytest-5356, pytest-dev__pytest-5404, pytest-dev__pytest-5413, pytest-dev__pytest-5479, pytest-dev__pytest-5495, pytest-dev__pytest-5559, pytest-dev__pytest-5787, pytest-dev__pytest-5840, pytest-dev__pytest-5980, pytest-dev__pytest-6116, pytest-dev__pytest-6186, pytest-dev__pytest-6197, pytest-dev__pytest-6214, pytest-dev__pytest-6283, pytest-dev__pytest-6323, pytest-dev__pytest-6368, pytest-dev__pytest-7046, pytest-dev__pytest-7122, pytest-dev__pytest-7158, pytest-dev__pytest-7168, pytest-dev__pytest-7186, pytest-dev__pytest-7231, pytest-dev__pytest-7283, pytest-dev__pytest-7314, pytest-dev__pytest-7324, pytest-dev__pytest-7481, pytest-dev__pytest-7490, pytest-dev__pytest-7499, pytest-dev__pytest-7500, pytest-dev__pytest-7521, pytest-dev__pytest-7637, pytest-dev__pytest-7648, pytest-dev__pytest-7985, pytest-dev__pytest-8055, pytest-dev__pytest-8124, pytest-dev__pytest-8365, pytest-dev__pytest-8428, pytest-dev__pytest-8447, pytest-dev__pytest-8463, pytest-dev__pytest-8516, pytest-dev__pytest-8906, pytest-dev__pytest-8950, pytest-dev__pytest-9064, pytest-dev__pytest-9249, pytest-dev__pytest-9279, pytest-dev__pytest-9359, pytest-dev__pytest-9624, pytest-dev__pytest-9646, pytest-dev__pytest-9681, pytest-dev__pytest-9709, pytest-dev__pytest-9780, pytest-dev__pytest-9911, pytest-dev__pytest-9956, scikit-learn__scikit-learn-10306, scikit-learn__scikit-learn-10331, scikit-learn__scikit-learn-10377, scikit-learn__scikit-learn-10382, scikit-learn__scikit-learn-10397, scikit-learn__scikit-learn-10427, scikit-learn__scikit-learn-10428, scikit-learn__scikit-learn-10443, scikit-learn__scikit-learn-10452, scikit-learn__scikit-learn-10471, scikit-learn__scikit-learn-10483, scikit-learn__scikit-learn-10495, scikit-learn__scikit-learn-10508, scikit-learn__scikit-learn-10558, scikit-learn__scikit-learn-10577, scikit-learn__scikit-learn-10774, scikit-learn__scikit-learn-10777, scikit-learn__scikit-learn-10881, scikit-learn__scikit-learn-10899, scikit-learn__scikit-learn-10913, scikit-learn__scikit-learn-10949, scikit-learn__scikit-learn-10982, scikit-learn__scikit-learn-11040, scikit-learn__scikit-learn-11042, scikit-learn__scikit-learn-11043, scikit-learn__scikit-learn-11151, scikit-learn__scikit-learn-11206, scikit-learn__scikit-learn-11235, scikit-learn__scikit-learn-11264, scikit-learn__scikit-learn-11315, scikit-learn__scikit-learn-11391, scikit-learn__scikit-learn-11496, scikit-learn__scikit-learn-11542, scikit-learn__scikit-learn-11574, scikit-learn__scikit-learn-11585, scikit-learn__scikit-learn-11596, scikit-learn__scikit-learn-11635, scikit-learn__scikit-learn-12258, scikit-learn__scikit-learn-12421, scikit-learn__scikit-learn-12443, scikit-learn__scikit-learn-12462, scikit-learn__scikit-learn-12486, scikit-learn__scikit-learn-12557, scikit-learn__scikit-learn-12626, scikit-learn__scikit-learn-12656, scikit-learn__scikit-learn-12682, scikit-learn__scikit-learn-12733, scikit-learn__scikit-learn-12758, scikit-learn__scikit-learn-12784, scikit-learn__scikit-learn-12827, scikit-learn__scikit-learn-12860, scikit-learn__scikit-learn-12908, scikit-learn__scikit-learn-12938, scikit-learn__scikit-learn-12961, scikit-learn__scikit-learn-12983, scikit-learn__scikit-learn-12989, scikit-learn__scikit-learn-13010, scikit-learn__scikit-learn-13013, scikit-learn__scikit-learn-13046, scikit-learn__scikit-learn-13087, scikit-learn__scikit-learn-13143, scikit-learn__scikit-learn-13157, scikit-learn__scikit-learn-13165, scikit-learn__scikit-learn-13174, scikit-learn__scikit-learn-13253, scikit-learn__scikit-learn-13283, scikit-learn__scikit-learn-13302, scikit-learn__scikit-learn-13313, scikit-learn__scikit-learn-13333, scikit-learn__scikit-learn-13363, scikit-learn__scikit-learn-13368, scikit-learn__scikit-learn-13392, scikit-learn__scikit-learn-13436, scikit-learn__scikit-learn-13536, scikit-learn__scikit-learn-13549, scikit-learn__scikit-learn-13554, scikit-learn__scikit-learn-13618, scikit-learn__scikit-learn-13628, scikit-learn__scikit-learn-13641, scikit-learn__scikit-learn-13780, scikit-learn__scikit-learn-13828, scikit-learn__scikit-learn-13877, scikit-learn__scikit-learn-13910, scikit-learn__scikit-learn-13915, scikit-learn__scikit-learn-13933, scikit-learn__scikit-learn-13960, scikit-learn__scikit-learn-13974, scikit-learn__scikit-learn-14012, scikit-learn__scikit-learn-14024, scikit-learn__scikit-learn-14125, scikit-learn__scikit-learn-14237, scikit-learn__scikit-learn-14430, scikit-learn__scikit-learn-14464, scikit-learn__scikit-learn-14520, scikit-learn__scikit-learn-14544, scikit-learn__scikit-learn-14591, scikit-learn__scikit-learn-14629, scikit-learn__scikit-learn-14704, scikit-learn__scikit-learn-14706, scikit-learn__scikit-learn-14806, scikit-learn__scikit-learn-14878, scikit-learn__scikit-learn-14898, scikit-learn__scikit-learn-14983, scikit-learn__scikit-learn-14999, scikit-learn__scikit-learn-15028, scikit-learn__scikit-learn-15084, scikit-learn__scikit-learn-15086, scikit-learn__scikit-learn-15094, scikit-learn__scikit-learn-15120, scikit-learn__scikit-learn-15138, scikit-learn__scikit-learn-15495, scikit-learn__scikit-learn-15524, scikit-learn__scikit-learn-23099, scikit-learn__scikit-learn-24145, scikit-learn__scikit-learn-24677, scikit-learn__scikit-learn-24769, scikit-learn__scikit-learn-25299, scikit-learn__scikit-learn-25308, scikit-learn__scikit-learn-25363, scikit-learn__scikit-learn-25500, scikit-learn__scikit-learn-25589, scikit-learn__scikit-learn-25601, scikit-learn__scikit-learn-25638, scikit-learn__scikit-learn-25672, scikit-learn__scikit-learn-25694, scikit-learn__scikit-learn-25697, scikit-learn__scikit-learn-25744, scikit-learn__scikit-learn-25747, scikit-learn__scikit-learn-25774, scikit-learn__scikit-learn-25805, scikit-learn__scikit-learn-25969, scikit-learn__scikit-learn-26194, scikit-learn__scikit-learn-26242, scikit-learn__scikit-learn-26289, scikit-learn__scikit-learn-26318, scikit-learn__scikit-learn-26400, scikit-learn__scikit-learn-26634, scikit-learn__scikit-learn-26644, scikit-learn__scikit-learn-3840, scikit-learn__scikit-learn-7760, scikit-learn__scikit-learn-9274, scikit-learn__scikit-learn-9775, scikit-learn__scikit-learn-9939, sphinx-doc__sphinx-10021, sphinx-doc__sphinx-10067, sphinx-doc__sphinx-10097, sphinx-doc__sphinx-10191, sphinx-doc__sphinx-10207, sphinx-doc__sphinx-10320, sphinx-doc__sphinx-10323, sphinx-doc__sphinx-10353, sphinx-doc__sphinx-10360, sphinx-doc__sphinx-10427, sphinx-doc__sphinx-10435, sphinx-doc__sphinx-10451, sphinx-doc__sphinx-10481, sphinx-doc__sphinx-10551, sphinx-doc__sphinx-10614, sphinx-doc__sphinx-10757, sphinx-doc__sphinx-10807, sphinx-doc__sphinx-10819, sphinx-doc__sphinx-11109, sphinx-doc__sphinx-11192, sphinx-doc__sphinx-11266, sphinx-doc__sphinx-11311, sphinx-doc__sphinx-11312, sphinx-doc__sphinx-11316, sphinx-doc__sphinx-11445, sphinx-doc__sphinx-11489, sphinx-doc__sphinx-11503, sphinx-doc__sphinx-11510, sphinx-doc__sphinx-11550, sphinx-doc__sphinx-7234, sphinx-doc__sphinx-7305, sphinx-doc__sphinx-7350, sphinx-doc__sphinx-7351, sphinx-doc__sphinx-7356, sphinx-doc__sphinx-7374, sphinx-doc__sphinx-7380, sphinx-doc__sphinx-7395, sphinx-doc__sphinx-7454, sphinx-doc__sphinx-7462, sphinx-doc__sphinx-7501, sphinx-doc__sphinx-7557, sphinx-doc__sphinx-7578, sphinx-doc__sphinx-7590, sphinx-doc__sphinx-7593, sphinx-doc__sphinx-7597, sphinx-doc__sphinx-7615, sphinx-doc__sphinx-7670, sphinx-doc__sphinx-7686, sphinx-doc__sphinx-7738, sphinx-doc__sphinx-7748, sphinx-doc__sphinx-7757, sphinx-doc__sphinx-7760, sphinx-doc__sphinx-7762, sphinx-doc__sphinx-7831, sphinx-doc__sphinx-7859, sphinx-doc__sphinx-7906, sphinx-doc__sphinx-7923, sphinx-doc__sphinx-7930, sphinx-doc__sphinx-7985, sphinx-doc__sphinx-8007, sphinx-doc__sphinx-8020, sphinx-doc__sphinx-8026, sphinx-doc__sphinx-8028, sphinx-doc__sphinx-8035, sphinx-doc__sphinx-8037, sphinx-doc__sphinx-8056, sphinx-doc__sphinx-8058, sphinx-doc__sphinx-8075, sphinx-doc__sphinx-8095, sphinx-doc__sphinx-8117, sphinx-doc__sphinx-8125, sphinx-doc__sphinx-8202, sphinx-doc__sphinx-8265, sphinx-doc__sphinx-8273, sphinx-doc__sphinx-8278, sphinx-doc__sphinx-8282, sphinx-doc__sphinx-8284, sphinx-doc__sphinx-8362, sphinx-doc__sphinx-8474, sphinx-doc__sphinx-8481, sphinx-doc__sphinx-8539, sphinx-doc__sphinx-8548, sphinx-doc__sphinx-8551, sphinx-doc__sphinx-8552, sphinx-doc__sphinx-8579, sphinx-doc__sphinx-8593, sphinx-doc__sphinx-8599, sphinx-doc__sphinx-8611, sphinx-doc__sphinx-8621, sphinx-doc__sphinx-8627, sphinx-doc__sphinx-8633, sphinx-doc__sphinx-8638, sphinx-doc__sphinx-8658, sphinx-doc__sphinx-8674, sphinx-doc__sphinx-8679, sphinx-doc__sphinx-8697, sphinx-doc__sphinx-8707, sphinx-doc__sphinx-8719, sphinx-doc__sphinx-8729, sphinx-doc__sphinx-8731, sphinx-doc__sphinx-8771, sphinx-doc__sphinx-8863, sphinx-doc__sphinx-8951, sphinx-doc__sphinx-9015, sphinx-doc__sphinx-9053, sphinx-doc__sphinx-9104, sphinx-doc__sphinx-9128, sphinx-doc__sphinx-9155, sphinx-doc__sphinx-9171, sphinx-doc__sphinx-9180, sphinx-doc__sphinx-9207, sphinx-doc__sphinx-9229, sphinx-doc__sphinx-9230, sphinx-doc__sphinx-9233, sphinx-doc__sphinx-9234, sphinx-doc__sphinx-9258, sphinx-doc__sphinx-9260, sphinx-doc__sphinx-9261, sphinx-doc__sphinx-9289, sphinx-doc__sphinx-9350, sphinx-doc__sphinx-9386, sphinx-doc__sphinx-9459, sphinx-doc__sphinx-9461, sphinx-doc__sphinx-9547, sphinx-doc__sphinx-9591, sphinx-doc__sphinx-9602, sphinx-doc__sphinx-9654, sphinx-doc__sphinx-9658, sphinx-doc__sphinx-9665, sphinx-doc__sphinx-9673, sphinx-doc__sphinx-9797, sphinx-doc__sphinx-9798, sphinx-doc__sphinx-9799, sphinx-doc__sphinx-9828, sphinx-doc__sphinx-9902, sphinx-doc__sphinx-9931, sphinx-doc__sphinx-9982, sphinx-doc__sphinx-9987, sphinx-doc__sphinx-9997, sphinx-doc__sphinx-9999, sympy__sympy-11232, sympy__sympy-11400, sympy__sympy-11787, sympy__sympy-11788, sympy__sympy-11818, sympy__sympy-11831, sympy__sympy-11862, sympy__sympy-11870, sympy__sympy-11897, sympy__sympy-11919, sympy__sympy-12088, sympy__sympy-12108, sympy__sympy-12144, sympy__sympy-12171, sympy__sympy-12194, sympy__sympy-12214, sympy__sympy-12236, sympy__sympy-12286, sympy__sympy-12301, sympy__sympy-12307, sympy__sympy-12419, sympy__sympy-12428, sympy__sympy-12454, sympy__sympy-12472, sympy__sympy-12489, sympy__sympy-12798, sympy__sympy-12812, sympy__sympy-12881, sympy__sympy-12945, sympy__sympy-13018, sympy__sympy-13043, sympy__sympy-13091, sympy__sympy-13146, sympy__sympy-13173, sympy__sympy-13177, sympy__sympy-13185, sympy__sympy-13198, sympy__sympy-13236, sympy__sympy-13259, sympy__sympy-13264, sympy__sympy-13265, sympy__sympy-13279, sympy__sympy-13286, sympy__sympy-13309, sympy__sympy-13346, sympy__sympy-13364, sympy__sympy-13369, sympy__sympy-13429, sympy__sympy-13437, sympy__sympy-13441, sympy__sympy-13551, sympy__sympy-13581, sympy__sympy-13619, sympy__sympy-13682, sympy__sympy-13744, sympy__sympy-13768, sympy__sympy-13773, sympy__sympy-13806, sympy__sympy-13808, sympy__sympy-13840, sympy__sympy-13852, sympy__sympy-13895, sympy__sympy-13903, sympy__sympy-13915, sympy__sympy-13962, sympy__sympy-13978, sympy__sympy-13988, sympy__sympy-14024, sympy__sympy-14031, sympy__sympy-14070, sympy__sympy-14082, sympy__sympy-14085, sympy__sympy-14166, sympy__sympy-14180, sympy__sympy-14207, sympy__sympy-14248, sympy__sympy-14308, sympy__sympy-14317, sympy__sympy-14333, sympy__sympy-14564, sympy__sympy-14575, sympy__sympy-14627, sympy__sympy-14821, sympy__sympy-15017, sympy__sympy-15085, sympy__sympy-15151, sympy__sympy-15198, sympy__sympy-15222, sympy__sympy-15231, sympy__sympy-15241, sympy__sympy-15273, sympy__sympy-15286, sympy__sympy-15304, sympy__sympy-15308, sympy__sympy-15320, sympy__sympy-15346, sympy__sympy-15446, sympy__sympy-15555, sympy__sympy-15586, sympy__sympy-15596, sympy__sympy-15599, sympy__sympy-15625, sympy__sympy-15635, sympy__sympy-15685, sympy__sympy-15933, sympy__sympy-15948, sympy__sympy-15970, sympy__sympy-15971, sympy__sympy-15976, sympy__sympy-16003, sympy__sympy-16052, sympy__sympy-16056, sympy__sympy-16085, sympy__sympy-16088, sympy__sympy-16106, sympy__sympy-16221, sympy__sympy-16281, sympy__sympy-16331, sympy__sympy-16334, sympy__sympy-16422, sympy__sympy-16437, sympy__sympy-16449, sympy__sympy-16474, sympy__sympy-16503, sympy__sympy-16527, sympy__sympy-16597, sympy__sympy-16601, sympy__sympy-16632, sympy__sympy-16637, sympy__sympy-16781, sympy__sympy-16792, sympy__sympy-16840, sympy__sympy-16858, sympy__sympy-16862, sympy__sympy-16864, sympy__sympy-16901, sympy__sympy-16906, sympy__sympy-16943, sympy__sympy-16963, sympy__sympy-17010, sympy__sympy-17038, sympy__sympy-17067, sympy__sympy-17103, sympy__sympy-17194, sympy__sympy-17223, sympy__sympy-17251, sympy__sympy-17271, sympy__sympy-17273, sympy__sympy-17288, sympy__sympy-17313, sympy__sympy-17340, sympy__sympy-17394, sympy__sympy-17512, sympy__sympy-17630, sympy__sympy-17653, sympy__sympy-17696, sympy__sympy-17720, sympy__sympy-17770, sympy__sympy-17809, sympy__sympy-17813, sympy__sympy-18030, sympy__sympy-18033, sympy__sympy-18062, sympy__sympy-18087, sympy__sympy-18109, sympy__sympy-18116, sympy__sympy-18130, sympy__sympy-18137, sympy__sympy-18168, sympy__sympy-18191, sympy__sympy-18198, sympy__sympy-18199, sympy__sympy-18200, sympy__sympy-18256, sympy__sympy-18351, sympy__sympy-18477, sympy__sympy-18478, sympy__sympy-18587, sympy__sympy-18605, sympy__sympy-18630, sympy__sympy-18633, sympy__sympy-18650, sympy__sympy-18667, sympy__sympy-18698, sympy__sympy-18728, sympy__sympy-18835, sympy__sympy-18903, sympy__sympy-18922, sympy__sympy-18961, sympy__sympy-19007, sympy__sympy-19040, sympy__sympy-19091, sympy__sympy-19093, sympy__sympy-19182, sympy__sympy-19201, sympy__sympy-19254, sympy__sympy-19487, sympy__sympy-19601, sympy__sympy-19713, sympy__sympy-19885, sympy__sympy-20049, sympy__sympy-20115, sympy__sympy-20131, sympy__sympy-20134, sympy__sympy-20169, sympy__sympy-20264, sympy__sympy-20322, sympy__sympy-20428, sympy__sympy-20438, sympy__sympy-20442, sympy__sympy-20476, sympy__sympy-20639, sympy__sympy-20691, sympy__sympy-20741, sympy__sympy-20916, sympy__sympy-21171, sympy__sympy-21259, sympy__sympy-21260, sympy__sympy-21271, sympy__sympy-21286, sympy__sympy-21370, sympy__sympy-21432, sympy__sympy-21436, sympy__sympy-21476, sympy__sympy-21527, sympy__sympy-21567, sympy__sympy-21586, sympy__sympy-21596, sympy__sympy-21612, sympy__sympy-21627, sympy__sympy-21769, sympy__sympy-21849, sympy__sympy-21864, sympy__sympy-21930, sympy__sympy-21931, sympy__sympy-21932, sympy__sympy-21952, sympy__sympy-22080, sympy__sympy-22098, sympy__sympy-22236, sympy__sympy-22383, sympy__sympy-22402, sympy__sympy-22740, sympy__sympy-22773, sympy__sympy-23021, sympy__sympy-23141, sympy__sympy-23191, sympy__sympy-23413, sympy__sympy-23560, sympy__sympy-23729, sympy__sympy-23808, sympy__sympy-24102, sympy__sympy-24353, sympy__sympy-24562, sympy__sympy-24909
| example_link | model | min_elo |
|---|---|---|
| scikit-learn__scikit-learn-11281 | 20240820_honeycomb | 1392.471 |
| pydata__xarray-6394 | 20240820_honeycomb | 1392.471 |
| sphinx-doc__sphinx-7268 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-13757 | 20240820_honeycomb | 1392.471 |
| django__django-13158 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-18744 | 20240820_honeycomb | 1392.471 |
| django__django-17084 | 20240820_honeycomb | 1392.471 |
| django__django-13743 | 20240820_honeycomb | 1392.471 |
| django__django-11612 | 20240820_honeycomb | 1392.471 |
| matplotlib__matplotlib-25775 | 20240820_honeycomb | 1392.471 |
| pylint-dev__pylint-5136 | 20240820_honeycomb | 1392.471 |
| django__django-10997 | 20240820_honeycomb | 1392.471 |
| django__django-13218 | 20240820_honeycomb | 1392.471 |
| matplotlib__matplotlib-23563 | 20240820_honeycomb | 1392.471 |
| django__django-11011 | 20240820_honeycomb | 1392.471 |
| django__django-12486 | 20240820_honeycomb | 1392.471 |
| sphinx-doc__sphinx-9281 | 20240820_honeycomb | 1392.471 |
| astropy__astropy-14371 | 20240820_honeycomb | 1392.471 |
| scikit-learn__scikit-learn-12583 | 20240820_honeycomb | 1392.471 |
| django__django-11185 | 20240820_honeycomb | 1392.471 |
| scikit-learn__scikit-learn-10198 | 20240820_honeycomb | 1392.471 |
| django__django-14954 | 20240820_honeycomb | 1392.471 |
| sphinx-doc__sphinx-8459 | 20240820_honeycomb | 1392.471 |
| scikit-learn__scikit-learn-10459 | 20240820_honeycomb | 1392.471 |
| pytest-dev__pytest-7352 | 20240820_honeycomb | 1392.471 |
| django__django-14017 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-13361 | 20240820_honeycomb | 1392.471 |
| django__django-11001 | 20240820_honeycomb | 1392.471 |
| django__django-10301 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-22005 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-13878 | 20240820_honeycomb | 1392.471 |
| matplotlib__matplotlib-20805 | 20240820_honeycomb | 1392.471 |
| django__django-12496 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-15609 | 20240820_honeycomb | 1392.471 |
| django__django-16315 | 20240820_honeycomb | 1392.471 |
| sympy__sympy-13615 | 20240820_honeycomb | 1392.471 |
| django__django-13121 | 20240820_honeycomb | 1392.471 |
| pytest-dev__pytest-5631 | 20240820_honeycomb | 1392.471 |
| scikit-learn__scikit-learn-13497 | 20240820_honeycomb | 1392.471 |
| sphinx-doc__sphinx-8969 | 20240820_honeycomb | 1392.471 |
| django__django-15128 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-13774 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| sympy__sympy-15523 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-13822 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| pylint-dev__pylint-7114 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| matplotlib__matplotlib-25667 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| matplotlib__matplotlib-21042 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| scikit-learn__scikit-learn-25733 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-12184 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| pallets__flask-4935 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-16693 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| matplotlib__matplotlib-23299 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-11149 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| astropy__astropy-13032 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-13925 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| sympy__sympy-22706 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| matplotlib__matplotlib-20584 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| astropy__astropy-14628 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-11333 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| sphinx-doc__sphinx-10449 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| scikit-learn__scikit-learn-25752 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| sphinx-doc__sphinx-9464 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| sympy__sympy-14038 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-14007 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| pytest-dev__pytest-10081 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-16454 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-15643 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| astropy__astropy-8263 | 20240721_amazon-q-developer-agent-20240719-dev | 1345.734 |
| django__django-14771 | 20240617_factory_code_droid | 1336.439 |
| pytest-dev__pytest-10051 | 20240617_factory_code_droid | 1336.439 |
| sphinx-doc__sphinx-7440 | 20240617_factory_code_droid | 1336.439 |
| sphinx-doc__sphinx-8506 | 20240617_factory_code_droid | 1336.439 |
| pytest-dev__pytest-6680 | 20240617_factory_code_droid | 1336.439 |
| sympy__sympy-13974 | 20240617_factory_code_droid | 1336.439 |
| pylint-dev__pylint-8281 | 20240617_factory_code_droid | 1336.439 |
| django__django-12855 | 20240617_factory_code_droid | 1336.439 |
| sympy__sympy-19016 | 20240617_factory_code_droid | 1336.439 |
| sympy__sympy-13031 | 20240617_factory_code_droid | 1336.439 |
| sympy__sympy-17655 | 20240617_factory_code_droid | 1336.439 |
| scikit-learn__scikit-learn-13467 | 20240617_factory_code_droid | 1336.439 |
| sphinx-doc__sphinx-10673 | 20240617_factory_code_droid | 1336.439 |
| pylint-dev__pylint-8312 | 20240617_factory_code_droid | 1336.439 |
| django__django-11216 | 20240617_factory_code_droid | 1336.439 |
| django__django-16053 | 20240617_factory_code_droid | 1336.439 |
| django__django-13128 | 20240617_factory_code_droid | 1336.439 |
| psf__requests-2931 | 20240617_factory_code_droid | 1336.439 |
| django__django-14733 | 20240617_factory_code_droid | 1336.439 |
| matplotlib__matplotlib-20488 | 20240617_factory_code_droid | 1336.439 |
| sympy__sympy-12529 | 20240617_factory_code_droid | 1336.439 |
| scikit-learn__scikit-learn-13704 | 20240617_factory_code_droid | 1336.439 |
| sphinx-doc__sphinx-8620 | 20240628_autocoderover-v20240620 | 1323.226 |
| sympy__sympy-21379 | 20240628_autocoderover-v20240620 | 1323.226 |
| matplotlib__matplotlib-25085 | 20240628_autocoderover-v20240620 | 1323.226 |
| scikit-learn__scikit-learn-10908 | 20240628_autocoderover-v20240620 | 1323.226 |
| matplotlib__matplotlib-23987 | 20240628_autocoderover-v20240620 | 1323.226 |
| sphinx-doc__sphinx-8291 | 20240628_autocoderover-v20240620 | 1323.226 |
| astropy__astropy-13390 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-11964 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-11490 | 20240628_autocoderover-v20240620 | 1323.226 |
| sympy__sympy-12270 | 20240628_autocoderover-v20240620 | 1323.226 |
| matplotlib__matplotlib-25499 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-11206 | 20240628_autocoderover-v20240620 | 1323.226 |
| psf__requests-1327 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-13620 | 20240628_autocoderover-v20240620 | 1323.226 |
| scikit-learn__scikit-learn-14450 | 20240628_autocoderover-v20240620 | 1323.226 |
| pydata__xarray-3151 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-17051 | 20240628_autocoderover-v20240620 | 1323.226 |
| matplotlib__matplotlib-13989 | 20240628_autocoderover-v20240620 | 1323.226 |
| astropy__astropy-14484 | 20240628_autocoderover-v20240620 | 1323.226 |
| scikit-learn__scikit-learn-13472 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-11294 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-14351 | 20240628_autocoderover-v20240620 | 1323.226 |
| django__django-11298 | 20240628_autocoderover-v20240620 | 1323.226 |
| sympy__sympy-13001 | 20240628_autocoderover-v20240620 | 1323.226 |
| sympy__sympy-18273 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-11334 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-11989 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-12613 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| astropy__astropy-14508 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pylint-dev__pylint-6937 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pytest-dev__pytest-7220 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-15044 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-17176 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pytest-dev__pytest-11178 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-11141 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| matplotlib__matplotlib-20826 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-13952 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pydata__xarray-4683 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pydata__xarray-2905 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pydata__xarray-4687 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-21101 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| matplotlib__matplotlib-14043 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| matplotlib__matplotlib-25332 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| scikit-learn__scikit-learn-15096 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-15442 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-20590 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-16858 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-13884 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-14976 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-10989 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| psf__requests-3738 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| matplotlib__matplotlib-23188 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-14309 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| matplotlib__matplotlib-25746 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| pytest-dev__pytest-8033 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-18211 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-13667 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| sympy__sympy-12906 | 20240620_sweagent_claude3.5sonnet | 1301.975 |
| django__django-16260 | 20240615_appmap-navie_gpt4o | 1215.660 |
| django__django-11053 | 20240615_appmap-navie_gpt4o | 1215.660 |
| django__django-12132 | 20240615_appmap-navie_gpt4o | 1215.660 |
| django__django-15630 | 20240615_appmap-navie_gpt4o | 1215.660 |
| django__django-16657 | 20240615_appmap-navie_gpt4o | 1215.660 |
| sympy__sympy-17239 | 20240615_appmap-navie_gpt4o | 1215.660 |
| matplotlib__matplotlib-24189 | 20240615_appmap-navie_gpt4o | 1215.660 |
| sphinx-doc__sphinx-8801 | 20240615_appmap-navie_gpt4o | 1215.660 |
| django__django-13165 | 20240615_appmap-navie_gpt4o | 1215.660 |
| sphinx-doc__sphinx-8120 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-11234 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-13757 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-12325 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| pydata__xarray-6601 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| matplotlib__matplotlib-22931 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| sympy__sympy-20565 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-12121 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| pylint-dev__pylint-4970 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| scikit-learn__scikit-learn-10803 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-13964 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-12517 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| psf__requests-2393 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-8961 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| sphinx-doc__sphinx-8435 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-14416 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-11166 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-10213 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| psf__requests-4356 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-12049 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-12394 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| sphinx-doc__sphinx-7975 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| django__django-12284 | 20240509_amazon-q-developer-agent-20240430-dev | 1197.752 |
| sympy__sympy-11794 | 20240402_sweagent_gpt4 | 1158.560 |
| pytest-dev__pytest-10552 | 20240402_sweagent_gpt4 | 1158.560 |
| django__django-14602 | 20240402_sweagent_gpt4 | 1158.560 |
| django__django-14599 | 20240402_sweagent_gpt4 | 1158.560 |
| matplotlib__matplotlib-22865 | 20240402_sweagent_gpt4 | 1158.560 |
| scikit-learn__scikit-learn-11160 | 20240402_sweagent_gpt4 | 1158.560 |
| django__django-16306 | 20240402_sweagent_gpt4 | 1158.560 |
| sympy__sympy-13798 | 20240402_sweagent_gpt4 | 1158.560 |
| django__django-15161 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-15607 | 20240728_sweagent_gpt4o | 1145.585 |
| sympy__sympy-19783 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-13689 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-11555 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-15102 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-14584 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-12091 | 20240728_sweagent_gpt4o | 1145.585 |
| astropy__astropy-14539 | 20240728_sweagent_gpt4o | 1145.585 |
| django__django-13556 | 20240402_sweagent_claude3opus | 1060.794 |
| scikit-learn__scikit-learn-13280 | 20240402_sweagent_claude3opus | 1060.794 |
| pytest-dev__pytest-9475 | 20240402_sweagent_claude3opus | 1060.794 |
| django__django-16366 | 20240402_rag_claude3opus | 845.862 |
| django__django-16902 | 20240402_rag_claude3opus | 845.862 |
| sphinx-doc__sphinx-7961 | 20240402_rag_claude3opus | 845.862 |
| django__django-12304 | 20240402_rag_claude3opus | 845.862 |
| django__django-15525 | 20240402_rag_claude3opus | 845.862 |
| pytest-dev__pytest-5550 | 20231010_rag_claude2 | 724.073 |
| sphinx-doc__sphinx-11544 | 20231010_rag_claude2 | 724.073 |
| pydata__xarray-4994 | 20231010_rag_claude2 | 724.073 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | acc | tau |
|---|---|---|
| django__django-9003 | 0.125 | -0.416 |
| pydata__xarray-5731 | 0.125 | -0.329 |
| sympy__sympy-15809 | 0.188 | -0.220 |
| pytest-dev__pytest-5550 | 0.062 | -0.166 |
| pydata__xarray-4994 | 0.062 | -0.166 |
| sphinx-doc__sphinx-11544 | 0.062 | -0.166 |
| pydata__xarray-7393 | 0.188 | -0.161 |
| scikit-learn__scikit-learn-13454 | 0.250 | -0.132 |
| django__django-15206 | 0.188 | -0.132 |
| django__django-15525 | 0.062 | -0.118 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum Elo to solve each problem.