Downloading the Data

  • Option 1: Direct download from the tasks page, in JSON lines format.
     

  • Option 2: Using the datasets library.

Making a Leaderboard Submission

  • Create a comma-separated values (CSV) file with the headers (Task, ID, Prediction), where each row represents one output.
    For example:









    We recommend using our conversion script to
     produce the CSV file from JSON prediction files to avoid discrepancies.
     

  • Login to the website (using your Google account is recommended).
     

  • Upload your CSV file via the submission page.
     

  • Within a few minutes, check your email for a confirmation message that your submission has been received.
     

  • Results will be sent by email within 24 hours. Valid public submissions will immediately appear on the leaderboard.

If you need any help, please reach out to scrolls-benchmark-contact@googlegroups.com

from datasets import load_dataset

scrolls_datasets = ["gov_report", "summ_screen_fd", "qmsum",

                    "narrative_qa", "qasper", "quality", "contract_nli"]

data = [load_dataset("tau/scrolls", dataset) for dataset in scrolls_datasets]

Task,ID,Prediction

qasper,8941956c4b67e2436bbaf372a120f358f50c377b,"English, German, French"

qasper,5b63fb32633223fa4ee214979860349242a11451,"sentiment classifiers"

...

quality,72790_5QFDYSRE_4,"No, he worked in the spaceport on Mercury."

...

summ_screen_fd,fd_Gilmore_Girls_01x13,"Rory's charity rummage sale is a disaste..."

...