An Evaluation queue allows for people to submit Synapse Files, Docker images, etc. for evaluation. They are designed to support open-access data analysis and modeling challenges in Synapse. This framework provides tools for administrators to collect and analyze data models from Synapse users created for a specific goal or purpose.
To create a queue, you must first create a Synapse project. To learn how to do so, please follow instructions here. An Evaluation Queue can take several parameters that you can use to fine tune it to your preferences. The minimum requirements to create a queue are:
Additionally, you can pass in an optional quota parameter; please note that these options are only possible to implement through the Python or R client, not the web. It can be configured with the following terms:
The example below shows how to create a queue using all of the parameters described:
An Evaluation Queue can have limits. Submission “rounds” (start date, round duration, and number of rounds) with an optional submission quota (maximum submissions per participant or team) can be defined for each queue. There is no way to configure the round or quota settings of an Evaluation Queue from the web. The Evaluation ID can be found under the Challenge tab of your project. Please note that a Challenge tab will not appear on your project until you have created a challenge (Tools > Run Challenge). In the case below, the evaluation queue id is
Using this value, we can configure the
quota parameters of this evaluation queue with the R or Python client.
Each Evaluation has its own sharing settings, which limit who can interact with the Evaluation and in what way:
To set the sharing setting, go to the Challenge tab and see your list of Evaluations. Click on the
Share button per Evaluation and share it with the Teams or individuals you would like.
All submissions of an Evaluation queue can be views through the through the use of a leaderboard. To learn how to create a wiki page, please visit here. Below are instructions on how to set up a leaderboard. You must know the evaluation Id to do so; see the section on how to “Configure an Evaluation Queue” for instructions on finding the evaluation Id.
Once you click on Leaderboard, you will have to input your own query statement such as
select * from evaluation_9610091. Remember, 9610091 should be replaced with your own evaluation Id. To view all the columns available, click Refresh Columns.
Clicking Refresh Columns will add these default columns.
If you are happy with your leaderboard configurations, save both the configurations and the wiki page to see the Leaderboard.
Try posting a question to our Forum.
Let us know what was unclear or what has not been covered. Reader feedback is key to making the documentation better, so please let us know or open an issue in our Github repository (Sage-Bionetworks/synapseDocs).