An Evaluation queue allows for people to submit Synapse
Docker images, etc. for evaluation. They are designed to support open-access data analysis and modeling challenges in Synapse. This framework provides tools for administrators to collect and analyze data models from Synapse users created for a specific goal or purpose.
To create a queue, you must first create a Synapse
Project or have edit permissions on an existing Project. To create a Synapse Project, follow the instructions on the Project and Data Management page.
Once you’ve created your project, navigate to it and add
/challenge to the url (e.g. www.synapse.org/#!Synapse:syn12345/challenge). Click Tools in the right corner and select Add Evaluation Queue.
An Evaluation queue can take several parameters that you can use to customize your preferences.
Optionally, you can restrict how things are submitted by using a quota.
An Evaluation queue can only have one quota. You may specify the length of time the queue is open, the start date, round duration, and number of rounds. These are required parameters. It is optional to set submission limit.
Each Evaluation has sharing settings, which limit who can interact with the Evaluation.
To set the sharing settings, go to the Challenge tab to view your list of Evaluations. Click on the Share button per Evaluation and share it with the teams or individuals you would like.
While there isn’t technically a way of “closing” an evaluation queue, there are multiple ways to discontinue submissions for users.
can submitpermissions to it. If you have the ability to modify the permissions of a queue, you will still be able to submit to the queue due to your
Any Synapse entity may be submitted to an Evaluation Queue.
In the R and Python examples, you need to know the ID of the evaluation queue. This ID must be provided to you by administrators of the queue.
The submission function takes two optional parameters:
team. Name can be provided to customize the submission. The submission name is often used by participants to identify their submissions. If a name is not provided, the name of the entity being submitted will be used. As an example, if you submit a File named testfile.txt, and the name of the submission isn’t specified, it will default to testfile.txt. Team names can be provided to recognize a group of contributors.
import synapseclient syn = synapseclient.login() evaluation_id = "9610091" my_submission_entity = "syn1234567" submission = syn.submit( evaluation = evaluation_id, entity = my_submission_entity, name = "My Submission", # An arbitrary name for your submission team = "My Team Name") # Optional, can also pass a Team object or id
library(synapser) synLogin() evaluation_id <- "9610091" my_submission_entity <- "syn1234567" submission <- synSubmit( evaluation = evaluation_id, entity = my_submission_entity, name = "My Submission", # An arbitrary name for your submission team = "My Team Name") # Optional, can also pass a Team object or id
Every submission you make to an Evaluation queue has a unique ID. This ID should not be confused with Synapse IDs which start with syn. All submissions have a
Navigate to a file in Synapse and click on Tools in the upper right-hand corner. Select Submit To Challenge to pick the challenge for your submission. Follow the provided steps to complete your submission.
Submissions can be viewed and shared with users through Submission Views creating dynamic leaderboards. Submission annotations can be added to a SubmissionStatus object and are automatically indexed in the view.
Navigate to the Tables tab and under the Table Tools menu in the upper right-hand select Add Submission View:
You can name the view, and select the evaluation queues to include in the scope.
You can add multiple evaluation queues to the scope:
During the creation process the default columns for a submission view will be included:
Selecting Add All Annotations will automatically include all the annotations found on the submissions in the scope as columns for the view:
Once created, a submission view can be embedded into a wiki page using the Synapse Table/View wiki widget:
You can input your own query statement such as
SELECT * FROM syn22155139 ORDER BY score DESC. Remember, 22155139 should be replaced with the id of the submission view:
You may embed a
Submit To Evaluation widget on a Wiki page to improve visibility of your Evaluation queue. The widget allows participants to submit to multiple Evaluation queues within a Project or a single Evaluation queue.
Currently, this Wiki widget is required to submit Synapse Projects to an Evaluation queue. Synapse Docker repositories can not be submitted through this widget.
The “Evaluation Queue unavailable message” is customizable. A queue may appear unavailable to a user if:
To learn how to create a Wiki page, please visit the Wikis article.