Date | Model | Contributors | #Params | Input Length | Score (Average) | GovRep (R1/R2/RL) | SumScr (R1/R2/RL) | QMSum (R1/R2/RL) | Qspr (F1) | Nrtv (F1) | QALT (EM-T/H) | CNLI (EM) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
04/27/2022 | LongT5 XL | LongT5 | 3B | 16K | 41.89 | 54.7/28.2/30.2 | 35.8/9.6/21.1 | 34.9/11.8/23.5 | 53.1 | 29.3 | 46.0/42.1 | 88.2 |
04/28/2022 | LongT5 Large | LongT5 | 770M | 16K | 40.47 | 54.2/27.8/29.8 | 35.6/9.2/21.2 | 35.1/12.0/23.3 | 52.3 | 27.2 | 40.6/38.6 | 87.3 |
04/28/2022 | LongT5 Base | LongT5 | 220M | 16K | 38.22 | 53.5/27.3/29.3 | 34.8/9.6/21.1 | 33.9/11.0/22.8 | 46.6 | 23.0 | 37.9/36.6 | 85.6 |
08/27/2022 | BART-large SLED | Ivgi et al., | 406M | 16K | 37.99 | 57.5/26.3/27.4 | 35.2/8.7/19.4 | 34.2/11.0/22.0 | 46.9 | 24.1 | 34.8/34.8 | 87.3 |
03/14/2022 | UL2 | Google Research | 20B | 2K | 37.87 | 53.6/26.1/28.8 | 32.9/7.8/19.4 | 31.1/8.5/20.4 | 37.6 | 24.2 | 45.8/40.7 | 88.7 |
01/01/2022 | LED Base | SCROLLS team | 162M | 16K | 29.16 | 56.2/26.6/28.8 | 24.2/4.5/15.4 | 25.1/6.7/18.8 | 26.6 | 18.5 | 25.8/25.4 | 71.5 |
01/01/2022 | BART Base | SCROLLS team | 139M | 1K | 29.01 | 47.9/18.6/22.7 | 27.2/4.9/16.7 | 30.2/8.7/20.7 | 26.3 | 15.4 | 26.0/25.9 | 77.4 |
01/07/2022 | Naive | SCROLLS team | - | - | 19.35 | 45.3/17.9/20.8 | 19.6/1.8/11.0 | 14.2/2.0/9.3 | 3.4 | 1.5 | 25.2/26.1 | 66.0 |
Click here for a downloadable version of the leaderboard.