Downloading the Data

  • Option 1: Direct download from the tasks page, in JSON lines format.

  • Option 2: Using the datasets library.

Making a Leaderboard Submission

  • Create a comma-separated values (CSV) file with the headers (Task, ID, Prediction), where each row represents one output.
    For example:

    We recommend using our conversion script to
     produce the CSV file from JSON prediction files to avoid discrepancies.

  • Login to the website (using your Google account is recommended).

  • Upload your CSV file via the submission page.

  • Within a few minutes, check your email for a confirmation message that your submission has been received.

  • Results will be sent by email within 24 hours. Valid public submissions will immediately appear on the leaderboard.

  • Each user is limited to 5 submissions per week and a total of 10 submissions per month.

If you need any help, please reach out to

from datasets import load_dataset

scrolls_datasets = ["gov_report", "summ_screen_fd", "qmsum",

                    "narrative_qa", "qasper", "quality", "contract_nli"]

data = [load_dataset("tau/scrolls", dataset) for dataset in scrolls_datasets]


qasper,8941956c4b67e2436bbaf372a120f358f50c377b,"English, German, French"

qasper,5b63fb32633223fa4ee214979860349242a11451,"sentiment classifiers"


quality,72790_5QFDYSRE_4,"No, he worked in the spaceport on Mercury."


summ_screen_fd,fd_Gilmore_Girls_01x13,"Rory's charity rummage sale is a disaste..."