--- language: "en" --- # Documentation ## Documentation * ### [Getting Started](https://docs.synapse.org/synapse-docs/getting-started.md) Welcome to the Synapse docs site! This site exists to help you make the most of your experience using Synapse. Whether you're new to Synapse, or an experienced... * ### [About Synapse](https://docs.synapse.org/synapse-docs/about-synapse.md) Synapse is a collaborative research platform that helps you and your team share, organize, and discuss your scientific research. Whether you are part of a smal... * ### [Synapse Governance](https://docs.synapse.org/synapse-docs/synapse-governance.md) Synapse governance is an essential component of the Synapse platform; it is a system of policies, procedures, and tools for managing and protecting data in Syn... * ### [Navigating Synapse](https://docs.synapse.org/synapse-docs/navigating-synapse.md) If you're new to Synapse, you may be looking for an overview of the site structure and components. Your main point of entry into Synapse activity will likely b... * ### [Sage Offerings](https://docs.synapse.org/synapse-docs/sage-offerings.md) Note: Are you preparing a data management plan as part of a grant proposal (for example for the NIH or NSF)? Sage can help you develop a data management and sh... * ### [Finding and Downloading Data](https://docs.synapse.org/synapse-docs/finding-and-downloading-data.md) You can find and download files from the Synapse web interface or by using one of the programmatic clients. If you are downloading files from the web, you can ... * ### [Uploading and Organizing Data](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data.md) Need to upload data to Synapse? This section will cover how to do so in an organized and efficient manner so that your data can be easily accessed, downloaded,... * ### [Curating Data](https://docs.synapse.org/synapse-docs/curating-data.md) While Synapse's organizational tools (folders, tables, etc.) are used for uploading and managing data, there are also tools for curating and viewing your own d... * ### [Citing Data](https://docs.synapse.org/synapse-docs/citing-data.md) Synapse has several useful tools to cite data so you can properly attribute your work. Mint a digital object identifier (DOI) to fully integrate publications a... * ### [Collaborating in Synapse](https://docs.synapse.org/synapse-docs/collaborating-in-synapse.md) Synapse is a collaborative platform. There are a couple useful features you can use to communicate with others on Synapse: discussion forums and teams. Discuss... * ### [API Clients and Documentation](https://docs.synapse.org/synapse-docs/api-clients-and-documentation.md) What Are API Clients? An API client is a tool that lets you talk to Synapse using your computer's "programming language" instead of clicking around in the web ... * ### [Challenges](https://docs.synapse.org/synapse-docs/challenges.md) Challenges are open science, collaborative competitions for evaluating and comparing computational algorithms or solutions to problems. Synapse can be used to ... * ### [Cloud Computing](https://docs.synapse.org/synapse-docs/cloud-computing.md) Synapse provides physical storage for files using Amazon S3, however, you can configure your own custom storage locations as well. For example, data files can ... * ### [Use Cases](https://docs.synapse.org/synapse-docs/use-cases.md) We've created this docs site to include all of the information and instructions you may need to effectively use Synapse. However, the information is separated ... * ### [Glossary](https://docs.synapse.org/synapse-docs/glossary.md) Browse the glossary to learn more about terms and definitions commonly used throughout Synapse. ACT Abbreviation for the Synapse Access and Compliance Team, a ... * ### [Help](https://docs.synapse.org/synapse-docs/help.md) This documentation site is designed to help you use Synapse. If you've explored the site and find you still have unanswered questions, try these other options:... * ### [Release Notes](https://docs.synapse.org/synapse-docs/release-notes.md) Welcome to Synapse Releases, here is where we'll update you with releases and news related to the Synapse Platform and Portals, roadmap updates, and new featur... * ### [Managing Your Account](https://docs.synapse.org/synapse-docs/managing-your-account.md) This page describes how to create, access, and manage your Synapse account, including authentication, account recovery, and security best practices. Anyone can... --- language: "en" --- # About Synapse [Synapse](http://www.synapse.org/) is a collaborative research platform that helps you and your team share, organize, and discuss your scientific research. Whether you are part of a small private team or global consortia, Synapse offers tools to help you make connections across datasets, code, and other insights. ## Who manages Synapse? Synapse was created and is managed by [Sage Bionetworks](https://sagebionetworks.org/) (that's us!) Sage is a nonprofit health research organization based in Seattle, Washington. We were founded in 2009 on the basis of open science and patient advocacy. We aim to promote reproducible research and responsible data sharing throughout the biomedical community. That's why we created Synapse. Ethical use of data is of the utmost importance to Sage, and we employ robust [governance](https://help.synapse.org/docs/Synapse-Governance.2004255211.html) policies and procedures to monitor compliance and ensure data privacy. In addition to, and in conjunction with Synapse, Sage operates multiple platforms and data portals that serve the current and future needs of our communities. | **Platforms** | **Community Portals** | |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Synapse](https://www.synapse.org/) A collaborative, open-source research platform that allows teams to share data, track analyses, and collaborate. | [dHealth (Digital Health) Knowledge Portal](https://dhealth.synapse.org/) Designed to enable the discovery and download of digital and mobile health data, tools, and benchmarked outcomes and digital biomarkers. | | [Challenge Platform](https://challenges.synapse.org/) 

An open-science, collaborative competition framework for evaluating and comparing computational algorithms. | [Cancer Complexity Knowledge Portal](https://cancercomplexity.synapse.org/) The NCI Division of Cancer Biology supports multiple research programs composed of interdisciplinary communities of scientists who aim to integrate approaches, data, and tools to address important questions in basic and translational cancer research. | | [Agora](https://agora.adknowledgeportal.org/) An interactive platform for visually exploring curated genomic analyses of Alzheimer's disease (AD), including a list of early AD candidate target nominations. | [AD Knowledge Portal](https://adknowledgeportal.synapse.org/) A platform for accessing data, analyses, and tools that the National Institute on Aging's Alzheimer's Disease Translational Research Program generates. | | | [NF Data Portal](https://nf.synapse.org/) Designed to help openly explore and share NF datasets, analysis tools, resources, and publications related to neurofibromatosis. | | | [PsychENCODE (PEC) Knowledge Portal](https://psychencode.synapse.org/) A platform designed to promote a community for sharing data from neuropsychiatric disease research. | ## How does Synapse help scientific research? Synapse is helping scientists in several ways: * **Building community around data sharing** : Synapse hosts many [research communities](https://www.synapse.org/#!StandaloneWiki:ResearchCommunities) and [open scientific resources](https://www.synapse.org/#!StandaloneWiki:OpenResearchProjects). It is a central place where researchers can come together to share data and collaborate. Data can be annotated and queried in one place, even if it is physically stored in different locations. * **Making research more reproducible**: Most research projects are complex and change over time. Synapse helps you track who performed what parts of an analysis. It also tracks when data was added or changed within a project, and it helps you understand how a dataset or an analysis evolved. These tools help you to publish reproducible work that can be used more easily by others. * **Benchmarking and challenges:** Tackling a complex problem alone can be difficult. Synapse hosts crowdsourced competitions, including [DREAM Challenges](http://dreamchallenges.org/), where many people come together to solve important computational problems or to compare analytical approaches. * **Protecting sensitive data:** Synapse is designed to keep sensitive data safe while still being shared responsibly. The platform offers built-in features to control who can access data and how that data can be used. Synapse is also backed by a [governance](https://help.synapse.org/docs/Synapse-Governance.2004255211.html) team who routinely monitor data use and who set policies and procedures to govern data access. * **Accessing data programmatically** : If you are a programmer, you can also access Synapse through a [REST API](https://rest-docs.synapse.org/rest/index.html), [Python client](https://python-docs.synapse.org/), [R client](https://r-docs.synapse.org/), or [command line client](https://python-docs.synapse.org/tutorials/command_line_client/). Accessing Synapse programmatically allows you to seamlessly integrate Synapse with your scientific computing and analytical workflows. --- language: "en" --- # Annotating Data With Metadata Annotations help users search for and find data, and they are a powerful tool used to systematically group and/or describe things in Synapse. Annotations are stored as key-value pairs in Synapse, where the key defines a particular aspect of your data (for example, species, assay, file format) and the value defines a variable that belongs to that category (mouse, RNAseq, .bam). You can use annotations to add additional information about a project, file, folder, table, or view. Annotations can be based on an existing ontology or controlled vocabulary, or can be created as needed and modified later as your metadata evolves. For example, if you have uploaded a collection of alignment files in the BAM file format from an RNA-sequencing experiment, each representing a sample and experimental replicate, you can use annotations to surface this information in a structured way. Sometimes, users encode this information in file names, e.g., `sampleA_conditionB.bam`, which makes it "human-readable" but not searchable. In this case, you may want to add annotations that look like this: ![image-20221216-235824.png](https://docs.synapse.org/__attachments/a_34c00a6f3888185670082a7e076320e987fd13802351f037139079370b5a556e/image-20221216-235824.png?cb=1b4d495075389ede04178fa347db7641) You can add and edit annotations from the web or programmatically using the command line client, the [Python client](https://python-docs.synapse.org/reference/annotations/), the [R client](https://r-docs.synapse.org/articles/views.html#updating-annotations-using-view). Using the programmatic clients facilitates batching and automated population of annotations across many files. The web client can be used to bulk update many files using [views](https://docs.synapse.org/synapse-docs/views.md).Adding and Editing Annotations via the Synapse UI To add or modify annotations on projects, files, folders, or tables in the web client, find the **Tools** menu in the upper right corner and select **Annotations**. ![image-20230308-191249.png](https://docs.synapse.org/__attachments/a_48f0169c809e91698563f0f51777e789102ba60829246d0119679d67e2c4f34b/image-20230308-191249.png?cb=e6ddef1ba2517be74b4549993b72ad41) A new window will appear with a list of any previously added annotations. To add new annotations or edit existing annotations, click **Edit**. ![image-20221216-235501.png](https://docs.synapse.org/__attachments/a_0a858142af0c51278057bf0759a6d22a46aff697e56a542485f2f41cef66cee4/image-20221216-235501.png?cb=0d1f2ed5740ab084bcd611462741542f) In the pop-up window, add your annotations one at a time. Use the **+** icon to add multiple values for a single key and the **x** icon to remove values. Click **Add New Key** to add a new key. ![image-20221216-235549.png](https://docs.synapse.org/__attachments/a_b08013577219a2d4e83ed6ce58803740aff6b1bde1ec22ad1f359f33b3a9d4d2/image-20221216-235549.png?cb=361589fb8fac882ccd1591c5bd0d3812) ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To add annotations on multiple files, refer to [Managing Custom Metadata at Scale](https://docs.synapse.org/synapse-docs/managing-custom-metadata-at-scale.md) for a tutorial on using [views](https://docs.synapse.org/synapse-docs/views.md) for annotation management. ## Adding and Editing Annotations Programmatically You can programmatically add annotations during file upload or after. **Command line** To add annotations on a new file during upload: synapse store sampleA_conditionB.bam --parentId syn00123 --annotations '{"fileFormat":"bam", "assay":"rnaSeq"}' To add annotations on an existing file: synapse set-annotations --id syn00123 --annotations '{"fileFormat":"bam", "assay":"rnaSeq"}' **Python** To add annotations on a new file during upload: from synapseclient.models import File entity = File( path="sampleA_conditionB.bam", parent_id="syn00123", annotations={"fileFormat":"bam", "assay":"rnaSeq"} ) entity.store() To modify annotations on an existing file: from synapseclient.models import File entity = File(id="123").get() entity.annotations['fileFormat'] = 'fastq' entity.store() **R** To add annotations on a new file during upload: entity <- File("sampleA_conditionB.bam", parent="syn00123") entity <- synStore(entity, annotations=list(fileFormat = "bam", assay = "rnaSeq")) To modify annotations on an existing file: entity <- synGet("syn00123") ##### Modify annotations and PRESERVE existing annotations entity$annotations$dataType = "foo" synStore(entity) --- language: "en" --- # API Clients and Documentation ## What Are API Clients? An **API client** is a tool that lets you talk to Synapse using your computer's "programming language" instead of clicking around in the web interface. Think of it like having a set of remote controls that can tell Synapse what to do---such as uploading, downloading, or organizing files---without you having to do each step manually. Instead of pointing and clicking in the Synapse website, an API client lets you type commands (or run scripts) that Synapse understands. This is especially helpful when you need to repeat the same action many times or work with a large amount of data. ## Why Use an API Client Instead of the Web Interface? While the Synapse website is great for browsing projects and making quick changes, there are times when it's not the most efficient or effective option. For example: * **Reproducibility of Research** By using a Synapse client, you can write scripts that perform your analysis or data processing steps. Sharing these scripts with others allows them to reproduce your work exactly---using the same data, same steps, and same outputs---helping ensure scientific transparency and trust. * **Large-Scale File Transfers** If you need to upload or download hundreds---or even thousands---of files, doing it one-by-one in the website would be slow and error-prone. An API client can handle big batches automatically. * **Automation and Integration** If you want Synapse to work as part of a bigger process---such as feeding data directly into an analysis pipeline---an API client allows you to connect Synapse to other tools and systems seamlessly. ## API Clients Available for Synapse ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to install Synapse API clients [here](https://docs.synapse.org/synapse-docs/installing-synapse-api-clients.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to manage stored login credentials [here](https://docs.synapse.org/synapse-docs/client-configuration.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Below, find links to Python, R, command line, and REST API documentation. 1. [Command Line Docs](https://python-docs.synapse.org/en/stable/tutorials/command_line_client/) : Connect and interact with Synapse directly using the command line client 2. [Python Docs](https://python-docs.synapse.org/): Interact with Synapse using the python client 3. [R Docs](https://r-docs.synapse.org/) : Use the R client to interact with Synapse from scripts or interactive R sessions 4. [REST API Docs](https://rest-docs.synapse.org/rest/) : Build your own client using the Synapse REST APIs A note about the R client We maintain the R client to support our R user community, recognizing R as a foundational language in many data workflows. That said, the R client is built on top of the Synapse Python client via the reticulate package. This design allows us to solve complex engineering problems---such as multi-threaded uploads/downloads and file caching---at the Python layer and reuse those solutions in R. While we aim to provide a reliable R experience, the dependency on Python introduces installation nuances, especially around environment management and package versions. If you encounter issues, we recommend using the Python client directly, which is actively developed and more broadly used. We appreciate your patience and continued feedback as we work to improve the user experience across both ecosystems. --- language: "en" --- # Challenges **Challenges** are open science, collaborative competitions for evaluating and comparing computational algorithms or solutions to problems. Synapse can be used to manage these scientific computing challenges. Any challenges that you are registered for will appear in the challenges tab of your [Synapse toolbar](https://docs.synapse.org/synapse-docs/navigating-synapse.md). Synapse enables you to create and manage a challenge, participate in a challenge, and/or submit Synapse files or Docker images for evaluation via evaluation queues. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about how to create and manage a challenge [here](https://docs.synapse.org/synapse-docs/running-a-challenge.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about how to download submissions [here](https://help.synapse.org/docs/Evaluating-Submissions.4156948561.html). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about how to register for and participate in a challenge [here](https://docs.synapse.org/synapse-docs/participating-in-a-challenge.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about evaluation queues [here](https://docs.synapse.org/synapse-docs/evaluation-queues.md). --- language: "en" --- # Citing Data Synapse has several useful tools to cite data so you can properly attribute your work. Mint a digital object identifier (DOI) to fully integrate publications and relevant evidence into your Synapse project, or set provenance to explicitly link GitHub code and other resources to data files. Tracking these steps directly in Synapse increases trust in the reliability of your data and analyses. ## Provenance The Synapse provenance system is one of many solutions you can use to make your work reproducible by you and others. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about Synapse provenance [here](https://docs.synapse.org/synapse-docs/provenance.md). ## Digital Object Identifiers (DOIs) DOIs can be used as a unique identifier and are available in Synapse for projects, files, folders, tables, views, and datasets. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about using DOIs in Synapse [here](https://docs.synapse.org/synapse-docs/digital-object-identifiers-dois.md). --- language: "en" --- # Client Configuration If you are here, it means you are trying to configure one of Synapse's API clients to use Synapse programmatically. There are multiple ways one can login to Synapse via the Python, R, command line clients. We recommend users choose the method that fits their workflow best. ## Prerequisites * Create a [Personal Access Token](https://help.synapse.org/docs/Managing-Your-Account.2055405596.html#ManagingYourAccount-PersonalAccessTokens) (**aka: Synapse Auth Token** ) obtained from [Synapse](http://synapse.org/) under your Settings. * Note that a token must minimally have the **view** scope to be used. * Include **Download** and **Modify** permissions if you are using the clients to follow any subsequent tutorials. * Once a personal access token has been created it can be used for any of the options below. ## One Time Login ### Python Use the [synapseclient.login](https://python-docs.synapse.org/en/stable/reference/client/#synapseclient.Synapse.login) function import synapseclient syn = synapseclient.login(authToken="authtoken") #returns Welcome, First Last! #### Command Line Client Use the `synapse login` command synapse login -p $MY_SYNAPSE_TOKEN #### R library(synapser) synLogin(authToken="authtoken") ## Use `.synapseConfig` Synapse configuration parameters for frequently used client-interactions can be set in a configuration file. By default, the file is in the user's home directory and is called `.synapseConfig`. For example, you can set: * A new cache location * Third party credentials to access files stored outside of Synapse (e.g. AWS-S3, etc.) * Your Synapse credentials, preferably in the form of an [access token](https://help.synapse.org/docs/Managing-Your-Account.2055405596.html#ManagingYourAccount-PersonalAccessTokens) Note the period at the beginning of the file name that makes it a hidden system file on Linux-like operating systems, since it will contain sensitive information. For writing code using the client that is easy to share with others, please do not include your credentials in the code. Instead, please use the `~/.synapseConfig` file to manage your credentials. * Please read this [comprehensive guide](https://python-docs.synapse.org/en/stable/tutorials/authentication/#use-synapseconfig) to learn how to manage your Synapse Config file with the Python/CLI client. * Please read this [guide](https://r-docs.synapse.org/articles/manageSynapseCredentials.html#use--synapseconfig) to learn how to manage your Synapse Config file with the R client. Note: The Python/CLI client currently supports multi-config profiles within the Synapse Config file which is not supported within the R client, but the config file written for the R client will remain to be supported by the Python client. After you set up your synapse config, the following login commands should work without specifying your credentials within the command. ### Python import synapseclient syn = synapseclient.login() #returns Welcome, First Last! #### Command Line Client Use the `synapse login` command synapse login #### R library(synapser) synLogin() ## Use Environment Variable Setting the `SYNAPSE_AUTH_TOKEN` environment variable will allow you to login to Synapse with a [Personal Access Token](https://help.synapse.org/docs/Managing-Your-Account.2055405596.html#ManagingYourAccount-PersonalAccessTokens) The environment variable will take priority over credentials in the user's `.synapseConfig` file. 1. Open your shell configuration file (e.g., `~/.bashrc`, `~/.zshrc`, or `~/.profile`) 2. Add and save this in the shell configuration file export SYNAPSE_AUTH_TOKEN='' 1. If you are using R, alternatively, you may save this environmental variable within the `.Renviron` file of your Rstudio project. More information [here](https://docs.posit.co/ide/user/ide/guide/environments/r/managing-r.html). 3. You will be able to log in like the commands above in the "Use `~/.synapseConfig`" section. Here are some [alternative ways](https://python-docs.synapse.org/en/stable/tutorials/authentication/#use-environment-variable) you can log in using the environmental variable for the Python and command line client. --- language: "en" --- # Cloud Computing Synapse provides physical storage for files using Amazon S3, however, you can configure your own custom storage locations as well. For example, data files can physically reside in your own S3 bucket, or a local file server using a proxy servers. Creating a custom storage location allows you greater ownership and control of your files, especially when you have a large amount of data or when additional restrictions need to be set on the data. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to take advantage of custom storage locations [here](https://docs.synapse.org/synapse-docs/custom-storage-locations.md). *** ** * ** *** Using AWS Security Token Service (STS), Synapse can securely grant you temporary AWS credentials to access data directly in S3. This can be useful if you want to: * Download data in bulk * Upload data in bulk * Allow your compute cluster to read S3 objects using the S3 APIs All of which you can now do with minimal overhead from Synapse. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to compute directly on data in Synapse or S3 [here](https://docs.synapse.org/synapse-docs/compute-directly-on-data-in-synapse-or-s3.md). --- language: "en" --- # Collaborating in Synapse Synapse is a collaborative platform. There are a couple useful features you can use to communicate with others on Synapse: discussion forums and teams. ## Discussion Forums Discussion forums are a space to communicate with others, similar to a message board. The discussion forum is visible to users who have access to the project. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about discussion forums [here](https://docs.synapse.org/synapse-docs/discussion-forums.md). ## Teams Teams allow you to easily manage groups of users to control access to projects, communicate with colleagues, and participate in challenges. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about teams [here](https://docs.synapse.org/synapse-docs/teams.md). --- language: "en" --- # Combining Data from Multiple Table Sources *Database normalization* (not to be confused with *statistical normalization)* is a common technique used in data management to limit the amount of redundant data that is stored in a database, which can reduce the risk of introducing errors or inconsistencies in data, and simplify the process of updating data. When data has been structured in this way, it is considered to be *normalized* . However, for practical purposes, like presenting a report or running an analysis, data from normalized sources often must be joined together, or *denormalized*. If you store normalized data in Synapse [tables](https://docs.synapse.org/synapse-docs/organizing-data-with-tables.md), [views](https://docs.synapse.org/synapse-docs/views.md), or [datasets](https://docs.synapse.org/synapse-docs/datasets.md), you can combine separate data sources using **Materialized Views**. A materialized view is a type of Synapse table that is defined using a Synapse SQL statement, which can contain SQL keywords such as JOIN and UNION to combine existing Synapse tables. ## Creating a Materialized View You can create a materialized view from the "Tables" tab of any project in which you have permission to create new objects. From the toolbar of the Tables tab, click "Add New...", then click "Add Materialized View". Enter the name of your new materialized view, and enter a Synapse SQL query that will define the results of your materialized view. ### Allowed SQL Keywords In the defining SQL of a materialized view, you can use [any supported Synapse SQL you can use in a query](https://sagebionetworks.jira.com/wiki/spaces/DOCS/pages/2656305182), as well as the following operations: * LEFT JOIN * RIGHT JOIN * INNER JOIN * UNION (column definitions must match) Like other tables and views, JOINs and UNIONs are not permitted in a query on a materialized view. ### Example: Creating and Joining Denormalized Datasets Suppose we wanted to augment the classic [*Iris*](https://en.wikipedia.org/wiki/Iris_flower_data_set) dataset with the iris species' common name. Naïvely, we could just add another column to the dataset: | sepallength | sepalwidth | petallength | petalwidth | species | **commonName** | |-------------|------------|-------------|------------|------------|----------------------| | 5.1 | 3.5 | 1.4 | 0.2 | Setosa | bristle-pointed iris | | 4.9 | 3.0 | 1.4 | 0.2 | Setosa | bristle-pointed iris | | 7.0 | 3.2 | 4.7 | 1.4 | Versicolor | blue flag | | 6.4 | 3.2 | 4.5 | 1.5 | Versicolor | blue flag | | 6.3 | 3.3 | 6.0 | 2.5 | Virginica | Virginia blueflag | | 5.8 | 2.7 | 5.1 | 1.9 | Virginica | Virginia blueflag | However, this data is not normalized. Instead, we could create a table that maps the unique species name to its common name: | species | commonName | |------------|----------------------| | Setosa | bristle-pointed iris | | Versicolor | blue flag | | Virginica | Virginia blueflag | If we create two separate Synapse tables for the Iris dataset ([syn51941633](https://www.synapse.org/#!Synapse:syn51941633)) and the mapping between species and common name ([syn51941637](https://www.synapse.org/#!Synapse:syn51941637/tables/)), we can create a [new materialized view](https://www.synapse.org/#!Synapse:syn51941637/tables/) with the following defining SQL: SQL SELECT iris.sepallength as sepalLength, iris.sepalwidth as sepalWidth, iris.petallength as petalLength, iris.petalwidth as petalWidth, iris.species as species, nameMap.commonName as commonName FROM syn51941633 iris LEFT JOIN syn51941635 nameMap ON iris.species = nameMap.species Synapse will build the materialized view containing the following sample data, which can be further queried and treated like any other table or view: | sepalLength | sepalWidth | petalLength | petalWidth | species | commonName | |-------------|------------|-------------|------------|------------|----------------------| | 5.1 | 3.5 | 1.4 | 0.2 | Setosa | bristle-pointed iris | | 4.9 | 3.0 | 1.4 | 0.2 | Setosa | bristle-pointed iris | | 7.0 | 3.2 | 4.7 | 1.4 | Versicolor | blue flag | | 6.4 | 3.2 | 4.5 | 1.5 | Versicolor | blue flag | | 6.3 | 3.3 | 6.0 | 2.5 | Virginica | Virginia blueflag | | 5.8 | 2.7 | 5.1 | 1.9 | Virginica | Virginia blueflag | ## Querying Materialized Views Materialized views can be queried like any other table or view in Synapse. For more information, see [Querying Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/querying-tables-views-and-datasets.md) . ## Versioning Materialized Views Unlike tables and file views, you cannot create a snapshot of a materialized view. When the referenced tables and views are updated, the query results of your materialized view may change. One technique that can be used to ensure your materialized view results do not change unexpectedly is to reference snapshot versions of sources in the materialized view's defining SQL. See [Versioning Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/versioning-tables-views-and-datasets.md) for guidance on versioning source tables. ## Permissions on Materialized Views Any Synapse user with the view permission on a materialized view can query it. However, the results of a materialized view may vary depending on permissions in source tables and views referenced in the materialized view's defining SQL. For more information about permissions in Synapse, see [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md) . To query a materialized view, a Synapse user must have "download" permission on all source tables (if any exist). Additionally, each row in a materialized view that is derived from a row in a view is only visible to Synapse users that have view permission on the corresponding object, such as a file. Rows that are derived from multiple objects in one or more views are only visible if the user has view permission on all referenced objects. --- language: "en" --- # Compute Directly on Data in Synapse or S3 Using AWS Security Token Service (STS), Synapse can securely grant you temporary AWS credentials to access data directly in S3. This can be useful if you want to: * Download data in bulk * Upload data in bulk * Allow your compute cluster to read S3 objects using the S3 APIs All of which you can now do with minimal overhead from Synapse. There are a few important considerations when determining whether to enable STS on Synapse managed storage compared to external storage with an S3 bucket. With Synapse managed storage, permissions to access are read-only, thus data is only accessible to download or compute on directly once STS is enabled. Alternatively, read-only and read-write permissions can be granted on external storage allowing for data to be manipulated directly with the AWS command line interface. Subsequently, connections to Synapse can be updated if data is changed. In some cases, this workflow is preferable. **Note:** You can only create STS storage locations on an empty folder. This is to ensure consistency between the STS storage location and the Synapse folder. To use STS on existing data, see the Migrating Your Data section below. ## Synapse-Managed STS Storage Locations You can create an STS storage location using Synapse storage. Temporary S3 credentials will grant you access to the files and folders scoped to your STS storage location. **Note:** For Synapse storage, you can request read-only permissions through STS, but not write permissions. Therefore, you can only upload or modify existing files in Synapse storage through the Synapse website or clients. To set up the STS storage location using Synapse storage, first make sure you have an empty Synapse folder. Note that you will need write access to that folder. Then run the following code: **Python** Python # Set storage location import synapseclient import json syn = synapseclient.login() FOLDER = 'syn12345' destination = {'uploadType':'S3', 'stsEnabled':True, 'concreteType':'org.sagebionetworks.repo.model.project.S3StorageLocationSetting'} destination = syn.restPOST('/storageLocation', body=json.dumps(destination)) project_destination ={'concreteType': 'org.sagebionetworks.repo.model.project.UploadDestinationListSetting', 'settingsType': 'upload'} project_destination['locations'] = [destination['storageLocationId']] project_destination['projectId'] = FOLDER project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination)) **R** R #set storage location library(synapser) library(rjson) synLogin() folderId <- 'syn12345' destination <- list(uploadType='S3', stsEnabled=TRUE, concreteType='org.sagebionetworks.repo.model.project.S3StorageLocationSetting') destination <- synRestPOST('/storageLocation', body=toJSON(destination)) projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting', settingsType='upload') projectDestination$locations <- list(destination$storageLocationId) projectDestination$projectId <- folderId projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination)) Once the Synapse managed STS storage location is set up, you can upload files through the [Synapse](https://www.synapse.org/) or the Synapse client of your choice. ## External STS Storage Locations You can also create an STS storage location in an external AWS S3 bucket. There are benefits of creating connections to Synapse from an external bucket. If you already have data stored in S3, or if you have large amounts of data that you want to transfer with the AWS command line interface, you can avoid uploading data to Synapse-managed storage by creating connections directly to the S3 bucket. Enabling an STS storage location in the external bucket allows access to the S3 directly for future computing. Follow the steps in the [Custom storage locations](https://docs.synapse.org/synapse-docs/custom-storage-locations.md) article to set read-write or read-only permissions on your external S3 bucket and enable cross-origin resource sharing (CORS). You may use AWS Cloudformation for set up. Again, you will need an empty Synapse folder, and you will need write access to the Synapse folder. Instead of setting the S3 bucket as upload location, complete set up by running the following code on your Synapse folder: **Important:** If a baseKey is not specified, the temporary AWS credentials vended by STS will give users access to the whole bucket. To prevent access to the whole bucket, enter a folder path in your bucket that all files in the storage location should go into as the baseKey. **Python** # Set storage location import synapseclient import json syn = synapseclient.login() FOLDER = 'syn12345' destination = {'uploadType':'S3', 'stsEnabled':True, 'bucket':'nameofyourbucket', 'baseKey':'nameofyourbasekey', 'concreteType':'org.sagebionetworks.repo.model.project.ExternalS3StorageLocationSetting'} destination = syn.restPOST('/storageLocation', body=json.dumps(destination)) project_destination ={'concreteType': 'org.sagebionetworks.repo.model.project.UploadDestinationListSetting', 'settingsType': 'upload'} project_destination['locations'] = [destination['storageLocationId']] project_destination['projectId'] = FOLDER project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination)) **R** #set storage location library(synapser) library(rjson) synLogin() folderId <- 'syn12345' destination <- list(uploadType='S3', stsEnabled=TRUE, bucket='nameofyourbucket', baseKey='nameofyourbasekey', concreteType='org.sagebionetworks.repo.model.project.ExternalS3StorageLocationSetting') destination <- synRestPOST('/storageLocation', body=toJSON(destination)) projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting', settingsType='upload') projectDestination$locations <- list(destination$storageLocationId) projectDestination$projectId <- folderId projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination)) Once your STS storage location is set up on your Synapse folder, you can add files through the [Synapse](https://www.synapse.org/) website or the Synapse client of your choice. If you plan to upload files directly to your S3 bucket, or if you already have files in your S3 bucket, you can add representations of those files to Synapse programmatically. Follow the [steps to add files in your S3 bucket to Synapse](https://docs.synapse.org/articles/custom_storage_location.html#adding-files-in-your-s3-bucket-to-synapse). **Note:** Synapse automatically generates folders for files uploaded through the Synapse website or clients. Files added directly to S3 may not match this folder structure. We recommend against mixing these two methods of adding files to prevent confusion in the folder structure. ## Obtaining Temporary S3 Credentials Once your STS storage location is set up, you can use Synapse to request temporary AWS credentials to access your data in S3 directly. These temporary credentials are active for 12 hours. To get temporary credentials, Python and Java code is provided below. The [REST interface](https://rest-docs.synapse.org/rest/GET/entity/id/sts.html) is also available to request temporary credentials. **Java** StsCredentials stsCredentials = synapseClient.getTemporaryCredentialsForEntity(folderEntityId, StsPermission.read_only); AWSCredentials awsCredentials = new BasicSessionCredentials(stsCredentials.getAccessKeyId(), stsCredentials.getSecretAccessKey(), stsCredentials.getSessionToken()); AWSCredentialsProvider awsCredentialsProvider = new AWSStaticCredentialsProvider(awsCredentials); AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(awsCredentialsProvider).build(); **Python** import synapseclient import boto3 syn = synapseclient.login() sts_credentials = syn.restGET(f"/entity/{FOLDER}/sts?permission=read_only") client = boto3.client( 's3', aws_access_key_id=sts_credentials['accessKeyId'], aws_secret_access_key=sts_credentials['secretAccessKey'], aws_session_token=sts_credentials['sessionToken'], ) ent = syn.get("syn12345", downloadFile=False) client.download_file(ent._file_handle['bucketName'], ent._file_handle['key'], ent.name) # ent._file_handle['bucketName'] -- The name of the Synapse bucket to download from. # ent._file_handle['key'] -- The path of the file in the Synapse bucket # ent.name -- The path to the file to download to. ## Migrating Existing Data ### From an Existing S3 Bucket If you have existing data in an S3 bucket, either stand-alone data or data from a previous Synapse project, you can use [this sample code](https://github.com/Sage-Bionetworks/Synapse-Repository-Services/blob/develop/client/sample-code/src/main/java/org/sagebionetworks/sample/sts/MigrateS3Bucket.java) to migrate your S3 data to a new Synapse folder with a STS storage location. **Note:** You'll need to create a new folder with an STS-enabled storage location, as per the instructions above. Additionally, you must be the owner of the storage location in Synapse. ### From a Project with Multiple Buckets If your Synapse project uses data from multiple S3 buckets, or if the data is in an S3 bucket you don't own, then you may need to download the data and re-upload it a new Synapse folder with a STS storage location. Use [this sample code](https://github.com/Sage-Bionetworks/Synapse-Repository-Services/blob/develop/client/sample-code/src/main/java/org/sagebionetworks/sample/sts/MigrateSynapseProject.java) to migrate the data in yourproject. **Note:** You'll need to create a new folder with an STS-enabled storage location, as per the instructions above. **Note:** This method may incur data transfer costs from S3. ## Additional Restrictions * STS storage locations can only be added to folders, not projects. * An STS storage location can only be added to, or removed from, empty folders. * An STS storage location cannot be added alongside other storage locations. * If a parent folder has an STS storage location, all sub-folders will have the same storage location and ACL as the parent. * A folder with an STS storage location can only contain files and other folders. * A file or folder in an STS storage location cannot be moved to a folder with a different storage location. * A file in an STS storage location must have a bucket and base key matching that storage location. * If an STS storage location is defined on a folder, that folder cannot be placed within another folder hierarchy that also defines an STS storage location (even if they are the same storage location). **See Also;** [Custom Storage Locations](https://docs.synapse.org/synapse-docs/custom-storage-locations.md) *** ** * ** *** **Need More Help?** Ask a question in the Synapse [Help Forum](https://www.synapse.org/#!SynapseForum:default). Your feedback is key to improving our documentation, so [contact us](mailto:synapseinfo@sagebase.org) if something is unclear or open an [issue](https://sagebionetworks.jira.com/secure/CreateIssue.jspa?issuetype=3&pid=12124). --- language: "en" --- # Creating and Managing Wikis **Important:** Wiki pages do not have specific conditions for use. Do not put any protected human data in Synapse Wikis. Wikis provide a space to write narrative content to describe a project or content within a project. Wikis are available in Synapse on projects, folders, and files. Every project has a separate Wiki tab where you can create pages and a hierarchy of sub-pages. Wiki pages can be written using plain text, basic HTML, or Markdown. If you choose Markdown to write your wiki, a formatting guide is available within the wiki editing window (it is also available as a wiki [here](https://www.synapse.org/#!Wiki:syn2467792/ENTITY/64247)). Useful shortcuts are also available in the wiki editor tool bar, including: heading, bold, italic, strike-through, code block, subscript and superscript. A wiki inherits its [sharing settings](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md) from the project, folder, or file it is associated with, and it cannot have separate sharing settings. Anyone with 'view' permissions or greater can see the content of the wiki page. When a project, folder or file is shared with the public, the associated wiki is visible to viewers who are not logged in to Synapse. You can create wikis using the Synapse UI or programatically. Like files, folders, and other objects in Synapse, wikis have a unique Synapse ID (synID) associated with them, and you can use this synID to reference them elsewhere. ## Creating, Editing, and Deleting a Wiki via the Synapse UI Within an existing project or a new one, click the **Wiki** tab. Navigate to the **Wiki Tools** menu and select the **Edit Project Wiki** . Content in this project wiki becomes your project's home page. You can add subpages to your project wiki that will become links on the left side of your home page (read more in the [subpages section below](https://help.synapse.org/docs/Creating-and-Managing-Wikis.1975746682.html#CreatingandManagingWikis-CreatingPageHierarchywithSubpages)). ![edit-wiki.png](https://docs.synapse.org/__attachments/a_c191792163f96dc3405b67ab4e4c81bdd22fc0b6180fd4454f7362055caa0dab/edit-wiki.png?cb=8611ae0fd21f96e1ba28b98329382357) To add a wiki to a folder or file, navigate to the folder or file and select **Edit Wiki** from the**Tools** menu. Content added to a wiki can be previewed before saving. To delete a wiki select **Delete Wiki Page** under the**Wiki Tools** menu. ## Creating a Wiki Programatically **Command line** The command line client does not support the creation of wiki content. **Python** import synapseclient syn = synapseclient.login() projWiki = Wiki(title='Data Summary', owner = myProj) markdownText = '''This is an example sentence.''' projWiki['markdown'] = markdownText projWiki = syn.store(myWiki) **R** library(synapser) synLogin() markdownText <- "This is example sentence." wiki <- Wiki(owner="myProj", title="Data summary", markdown=markdownText) wiki <- synStore(myWiki) ## Creating Page Hierarchy with Subpages A project wiki can have subpages, which will appear nested below the main wiki page. Creating subpages will also create a navigation bar on the left side of the screen that lists the wiki subpages in the order they were created. Links to wiki subpages include the project ID, the wiki path, and the ID of the wiki page (e.g. ). The subpage ID can be found in the browser URL bar when the subpage is viewed. To add a subpage, use the**Wiki Tools** menu and then click **Add Wiki Subpage** . You can edit this sub-page like any other wiki page through the **Wiki Tools** menu and **Edit Project Wiki**. ### Modifying Page Order By default, wiki pages are created at a single level, or without a page hierarchy. As your wiki evolves over time, you may want to change the order and hierarchy of pages. To do this, click the **Edit Order** button below the wiki page navigation bar to change both the order and hierarchy of pages within a particular level. Click **Edit Order** , and then select the page you want to move. Use the arrow buttons to move the page up or down in your page tree. Use the left and right arrows to nest subpages deeper or promote them higher in a page hierarchy. When you are finished, click **Done** to save your work. Note that only users with the ability to edit the wiki content have the ability to edit the wiki order. Changing the wiki order does not change the synID of the wiki page. ## Wiki Widgets Widgets are Synapse features that can be added to supplement any Markdown text and customize your wiki. These include inserting images, tagging individuals, querying tables, and more. Use the **Insert** menu at the top of the wiki editor window to select a widget. Each of these widgets is described in the following table. To edit widgets after they have been added to the wiki, use the **Edit Synapse Widget** button in the upper left hand corner of the wiki editing window. Click on the Markdown text for a widget, and then click **Edit Synapse Widget** to view the widget configuration window again. | **Widget** | **Description** | |-------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Attachment** | Attach a file from your local computer to the wiki. | | **Button Link** | Insert a button that links to content within Synapse or elsewhere. **Tip:** buttons can be colored purple by adding `'&highlight=true'` to the end of the widget markdown | | **Collapsible Section (Details/Summary)** | Insert a label or heading that you can click to expand and reveal additional text. | | **Entity List** | A list of Synapse folders, files or tables can be created by browsing to the Synapse location or searching by entity name or Synapse ID. The table lists entity name, date entity was created, who created it, and version and version notes (if selecting files, tables, or views). | | **File Preview** | Embeds a preview window for .csv, .txt, and image files | | **Genome Browser** | You can add a [Biodalliance genome browser](http://www.biodalliance.org/) using tracks from files uploaded to Synapse or from external sources. Choose between Human or Mouse and adjust your tracks for height and color. [See the Genome Browser section below](https://help.synapse.org/docs/Creating-and-Managing-Wikis.1975746682.html#CreatingandManagingWikis-GenomeBrowser) for more information. | | **Image** | Embeds an JPG, PNG, GIF or SVG. The image can be uploaded, from the web, or from a Synapse file. ***Note:*** Images file names that have certain words (ie, 'ad') could be blocked if you are using an adblocker. | | **Join Team Button** | Provide a button for people to join Synapse [teams](https://docs.synapse.org/synapse-docs/teams.md). | | **Link** | Insert a URL or Synapse ID linked to text. | | **Provenance Graph** | Embeds the provenance graph created for a file. | | **Reference** | Create a reference list by linking to papers using the References widget. | | **Simple Plot** | Insert a simple bar chart from a file or table. | | **Submit to Evaluation** | Create a button for users to submit their entries to a Synapse challenge. | | **Table: Paste tabular data** | A table can be created of any data by pasting tab delimited content into this widget window. | | **Table: Query on a Synapse Table/View** | Provides a query for any Synapse table or view and displays the information in the wiki. | | **Table of Contents** | Creates a content list that links to sections of the wiki based on headers and subheaders. | | **Team Badge** | Creates a link to the team profile. | | **Team Member Count** | Select a team to display the team member count. | | **Team Members List** | Select a team to display a table of the team member names, institutions, and Synapse email addresses. | | **User** | Tag a user by entering their Synapse username. You can also do this by typing '@' while editing a wiki and enter the Synapse username or part of their full name in the dialog that appears. | | **Video** | Video, Vimeo Video, and YouTube Video insert a video from various sources. | ### Widgets in Experimental Mode Some widgets are available only in experimental mode. These widgets are still under development but are available for public use. To access these widgets, navigate to the lower right hand corner of your screen in Synapse, and click **Experimental Mode Off** . Read the experimental mode statement and click **OK** . Navigate back to a wiki page, where a new **@ Insert** menu option will appear at the top of the wiki editing window with a list of widgets in development. Use these widgets with caution, as they may change in the future and data created using them could be lost during upgrade. ## Adding a Table or View to a Wiki Wiki table widgets allow you to embed tabular data into a wiki, enabling quick access to files or data. There are two ways to embed tabular data into a wiki: either paste the data directly, or select data from existing tables or file views in Synapse. ### Pasting Tabular Data To embed tabular data in a wiki page, navigate to the wiki and select **Wiki Tools** , then **Edit Wiki** . From the **Insert** menu in the wiki editing window, select **Table: Paste Tabular Data** . In the next window, paste tab delimited data and click **Save**. ### Selecting Data from a Synapse Table or View If you uploaded a table or created a view in Synapse, you can use the **Table: Query on a Synapse Table/View** widget to add the entire table or view to your wiki. You can also choose to add only a subset of your table or view using a SQL-like query. ### Focus Scope In the example below, a specific subset of data from a file view is embedded in a wiki. This file view contains both RNA-Seq and SNP genomic data, as depicted in the file view scope shown in the image below. The scope can be restricted to RNA-Seq data with [additional queries](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html) on the embedded table. This example will demonstrate how to highlight .fastq and .bam RNA-Seq files in an embedded wiki table, so that team members have a clear path to access files for reproducible studies. ![inPractice_studyContainer.png](https://docs.synapse.org/__attachments/a_0ec3905ac6879ed1fc8b504fba1e3455eba5a88d4330d63df1f262b591aa8f2d/inPractice_studyContainer.png?cb=0667a34d4220ec188f3cb272b297a1c0) ### Restricting a Query A table query is required to embed table or file view data into your wiki. In this example, the parent file view is a combination of multiple assay types, and the query is restricted to isolate relevant RNA-Seq entries. Additionally, the columns can be limited to ensure only imperative information is visible in the wiki. In this example query below, only the columns for ID, version, file format, assay type, genome build, and date modified will be displayed from `syn17097374`. From that data, only data where assay type is recorded as `rnaSeq` and file format is `bam` or `fastq` will be shown in the wiki table. SELECT id,currentVersion,fileFormat,assay,genomeBuild,modifiedOn FROM syn17097374 WHERE "assay" = 'rnaSeq' AND "fileFormat" = 'bam' OR "fileFormat" = 'fastq' ### Embedding a Table Navigate to **Wiki Tools** to **Edit Project Wiki** or to **Folder Tools** to **Edit Folder Wiki** . From the wiki editing window, select **Insert** and **Table:Query on a Synapse Table/View** to insert the intended file view query. The result is a direct link to relevant files from a wiki page. Add narrative text to provide additional information or context for this view. ![inPractice_embeddedView.png](https://docs.synapse.org/__attachments/a_0f2a050628bbd1dca7f75e9c9fd4ef44e672a568392b31c8ef600b07745ff062/inPractice_embeddedView.png?cb=db243132155e82342a8306c9b97c9f1a) ## Forms Forms collect data from Synapse users and put it into a Synapse table. Forms are currently available only in experimental mode as they are not yet ready for full release; please use wiki forms only with this caveat in mind. With forms, you can: * Create surveys to collect feedback * Crowdsource data collection * Allow others to contribute data through a user-friendly user interface ### Creating a Table In order to use forms, you'll need to create a table first. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about creating tables at [Organizing Data With Tables](https://docs.synapse.org/synapse-docs/organizing-data-with-tables.md). ### Activating and Deactivating Experimental Mode On the right side of the Synapse footer, you will see **Experimental Mode Off** . Click this text to read a statement about enabling experimental mode, and click **OK** if you agree to the terms. Doing so enables a small number of new widgets in our wiki pages, all available under under the **@ Insert** menu in the wiki editing window. You can also deactivate experimental mode at any time; click on **Experimental Mode On** to deactivate it and return to the default Synapse product. ### Creating a Form Once experimental mode has been activated, visit the wiki page where you'd like to insert the form. Click on **Edit Project Wiki** and select the **@ Insert** menu. Select **Synapse Form** from the menu and enter the table synID into the resulting pop-up window. The following Markdown text will be added to your wiki. ${synapseForm?tableId=syn123&successMessage=Your response has been recorded} Optionally, you can add a customized "success" message for users by editing the Markdown text. When a form is submitted, it will create a new row in a Synapse table. * Column names are used as question prompts. Each column will appear as a separate question for users to answer. * The possible values for an enum column type, for example, will be a represented by a multiple choice question. The form is also responsive, so it will render on both large and small screens. ### Allowing Others to Contribute Entering data into a form is equivalent to editing the data in a table, so users who wish to add data using the form will need to have Edit permissions on the table. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) See [Sharing settings](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md) for more information on controlling who can edit your table. ## Genome Browser ### Biodalliance Setup * Species = Supports human (hg19/Gr37) and mouse (mm10/Grch38) genomes * chr = Chromosome number * viewStart = Start position * viewEnd = End position ### Supported File Types #### **BIGWIG** * Source name = Track name * File = synID of a bigwig file (.bw, .bigWig) * Height = Height of the track (pixels) * Color = Color of the track #### **VCF/BED** **Note:** To embed vcf and bed files, the files must be in the correct format. Please view how to prepare vcf and bed files below if they are not in the right format. **VCF (.vcf.gz AND .vcf.gz.tbi)** * Source name = Track name * File = synID of a compressed vcf file (.vcf.gz) * Tabix file: synID of a tabix index file (.vcf.gz.tbi) * Height = Height of a datapoint glyph in the track (pixels) * Color = Color of the track **BED (.bed.gz AND .bed.gz.tbi)** * Source name = Track name * File = synID of a compressed vcf file (.vcf.gz) * Tabix file: synID of a tabix index file (.vcf.gz.tbi) * Height = Height of a datapoint glyph in the track (pixels) * Color = Color of the track ### Preparing VCF and BED Files 1. [Download](http://sourceforge.net/projects/samtools/files/tabix/) and build the [tabix and bgzip](http://www.htslib.org/doc/tabix.html) programs 2. Compress your .vcf or .bed file using bgzip (example.vcf -\> example.vcf.gz, example.bed -\> example.bed.gz) #vcf bgzip example.vcf #bed bgzip example.bed 3. Create a tabix index file for the bgzip-compressed vcf or bed (example.vcf.gz.tbi, example.bed.gz.tbi) #vcf tabix -p vcf example.vcf.gz #bed tabix -p bed example.bed.gz 4. Upload both onto Synapse and enjoy [Biodalliance](http://www.biodalliance.org/)! --- language: "en" --- # Curating Data While Synapse's [organizational tools](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data.md) (folders, tables, etc.) are used for uploading and managing data, there are also tools for curating and viewing your own data, as well as other data already stored in Synapse. A primary curation tool for your own data is creating metadata. This involves annotating your data with standardized information in order to give it context---*data about the data*, if you will. Metadata is what allows data in Synapse to be searchable, discoverable, accessible, re-usable, and understandable to you, members of your team, and to others, including those who were not involved in the data generation process. Metadata can be descriptive (i.e., the name of the file), administrative (i.e., provenance information), or research-based (i.e., information about the sampling and handling of data). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn all about creating and managing this metadata at [Annotating Data With Metadata](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md). *** ** * ** *** Other tools for curating data include views, datasets, and versioning. You can use a view to: * Search and query many files, tables, projects, and submissions at once * View and edit file or table annotations in bulk * Group or link files, tables, projects, or submissions together by their annotations ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn all about views [here](https://docs.synapse.org/synapse-docs/views.md). You can use a dataset to: * Collect and distribute a set of files generated from the same study or project * Create a single item to represent a group of files that exist across disparate projects or folders ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn all about datasets [here](https://docs.synapse.org/synapse-docs/datasets.md). Versioning is a way to save new copies of your work each time you make a change. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn all about versioning [here](https://docs.synapse.org/synapse-docs/versioning.md). --- language: "en" --- # Custom Storage Locations Synapse provides physical storage for files using Amazon S3, however, you can configure your own custom storage locations as well. For example, data files can physically reside in your own S3 bucket, Google Cloud Storage Bucket, or a local file server using a proxy servers. Creating a custom storage location allows you greater ownership and control of your files, especially when you have a large amount of data or when additional restrictions need to be set on the data. **Note:** System metadata, annotations, and provenance records are still stored in Synapse's S3 storage. ## Setting Up an External AWS S3 Bucket This article will describe two ways to setup an external AWS S3 bucket: * [Setup with AWS Console](https://docs.synapse.org/articles/custom_storage_location.html#setup-with-aws-console): Manual setup using the [AWS Console](https://console.aws.amazon.com/console/). * [Setup with AWS Cloudformation](https://docs.synapse.org/articles/custom_storage_location.html#setup-with-cloudformation): Automated setup using [AWS Cloudformation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html). To begin, follow the documentation on Amazon Web Service (AWS) site to [**Create a Bucket**](http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html). Buckets do not need to be located in the US. Make the following adjustments to customize the bucket to work with Synapse: * When the AWS instructions prompt you to `Create a Bucket - Select a Bucket Name and Region`, use a unique name. For example, `thisisthenameofmybucket`. * Select the newly created bucket and click the **Permissions** tab. * Select the **Bucket Policy** button and copy one of the below policies (read-only or read-write permissions). Change the name of `Resource` from "[synapse-share.yourcompany.com](http://synapse-share.yourcompany.com/)" to the name of your new bucket (twice) and ensure that the `Principal` is `"AWS":"325565585839"`. This is Synapse's account number. **Note:** Files in an external bucket will not be automatically added to Synapse. To add files to Synapse that are already in your bucket, see below. ### Read-Write Permissions To allow authorized Synapse users to upload data to your bucket, read-write permissions need to be set on that bucket so you can allow Synapse to upload and retrieve files: { "Statement": [ { "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Effect": "Allow", "Resource": "arn:aws:s3:::thisisthenameofmybucket", "Principal": { "AWS": "325565585839" } }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload" ], "Effect": "Allow", "Resource": "arn:aws:s3:::thisisthenameofmybucket/*", "Principal": { "AWS": "325565585839" } } ] } To register the bucket with Synapse, you also need to create an object that proves to the Synapse service that you own this bucket. This can be done by creating a file named [**owner.txt**](https://docs.synapse.org/assets/downloads/owner.txt) that contains a **line or comma separated** list of *user identifiers* that are allowed to register and upload to the bucket. Valid *user identifiers* are a numeric Synapse user ID or the numeric ID of a team that you are a member of. The ID of the user or the team can be obtained by navigating to the user profile or to the team page. The ID is the numeric value shown in the browser URL bar after the **Profile:** or **Team:** prefixes. ![browserUserId.png](https://docs.synapse.org/__attachments/a_4fbd3214228d50e96934cdbc65da51cf98e22e37b4d317ce6fd6611d48bb94d8/browserUserId.png?cb=ce72e9c2217411b064d09f3e3d86e79f) ![browserTeamId.png](https://docs.synapse.org/__attachments/a_569ce004eb2a534030b20c225f0acc268ff705b6de313d7c1d2ed2fee4886996/browserTeamId.png?cb=ea7f6e96e4e1f1e7c2282c95c1b5b0c1) You can upload the file with the Amazon Web Console or the [AWS command line client](https://aws.amazon.com/cli/). **Web** Navigate to your bucket on the Amazon Console and select **Upload** to upload your text file. ![uploadAWS.png](https://docs.synapse.org/__attachments/a_9137906445e92fa868c751f0609c7727880f139463def4324f4d00d99b9d9628/uploadAWS.png?cb=0766d40ad74d89f213ca418df2cd3cf4) **Command line** # copy your owner.txt file to your s3 bucket aws s3 cp owner.txt s3://nameofmybucket/nameofmyfolder ### Read-Only Permissions If you do not want to allow authorized Synapse users to upload data to your bucket but provide read access instead, you can change the permissions to read-only: { "Statement": [ { "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Effect": "Allow", "Resource": "arn:aws:s3:::synapse-share.yourcompany.com", "Principal": { "AWS": "325565585839" } }, { "Action": [ "s3:GetObject" ], "Effect": "Allow", "Resource": "arn:aws:s3:::synapse-share.yourcompany.com/*", "Principal": { "AWS": "325565585839" } } ] } ### Enable Cross-Origin Resource Sharing (CORS) In **Permissions** , click **CORS configuration** . In the CORS configuration editor, edit the configuration so that Synapse is included in the `AllowedOrigin` tag. An example CORS configuration that would allow this is: * GET POST PUT HEAD 3000 * For more information, please read: [How Do I Configure CORS on My Bucket?](https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html#how-do-i-enable-cors) ### Setup with AWS Cloudformation For convienance, [AWS Cloudformation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html) can be used to provision a custom AWS S3 bucket for use with Synapse. Using this approach will result in the exact same bucket as described in [Setup with AWS Console](https://docs.synapse.org/articles/custom_storage_location.html#setup-with-aws-console). Instructions: 1. Download the [CF template](https://docs.synapse.org/__attachments/a_ae83d8892175fb84169e9f37fe62a647bec6f6269ffa69ddf49694117d89f74a/synapse-s3-bucket_updated.yaml.md?cb=5c1a892fc197fe3473ff530e74a60f00). 2. Use the [AWS Command Line](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/index.html) or [AWS Console](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create) to execute the template which will automatically provision the bucket. Example using the the `awscli`: aws cloudformation create-stack \ --stack-name MyCustomSynapseBucket \ --template-body file://SynapseExternalBucket.yaml \ --parameters ParameterKey=Department,ParameterValue=Cancer ParameterKey=Project,ParameterValue=Mammography \ ParameterKey=OwnerEmail,ParameterValue=joe.smith@company.com ParameterKey=SynapseUserName,ParameterValue=jsmith The above example shows required parameters: * Department - A department tag. Can be any arbitrary text. * Project - A project tag. Can be any arbitrary text. * OwnerEmail - A bucket owner tag. A valid email. * SynapseUserName - The Synapse account user name. **Note** : Department, Project, OwnerEmail are only used to [tag the bucket](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-resource-tags.html) and can be arbitrary. The following are optional parameters: # (Optional) true for read-write, false (default) for read-only bucket AllowWriteBucket: 'true' # (Optional) true (default) to encrypt bucket, false for no encryption EncryptBucket: 'false' # (Optional) 'Enabled' to enable bucket versioning, default is 'Suspended' BucketVersioning: 'Enabled' # (Optional) 'Enabled' to enable bucket data life cycle rule, default is 'Disabled' EnableDataLifeCycle: 'Enabled' # (Optional) S3 bucket objects will transition into this storage class: GLACIER(default), STANDARD_IA, ONEZONE_IA LifecycleDataStorageClass: 'STANDARD_IA' # (Optional) Number of days until S3 objects are moved to the LifecycleDataStorageClass, default is 30 LifecycleDataTransition: '90' # (Optional) Number of days (from creation) when objects are deleted from S3 and LifecycleDataStorageClass, default is 365000 LifecycleDataExpiration: '1825' # (Optional) Restrict downloading files from this bucket to only AWS resources (e.g. EC2 , Lambda) within the same region as this bucket. default is false. SameRegionResourceAccessToBucket: 'true' After executing the Cloudformation command, view the [AWS Cloudformation dashboard](https://console.aws.amazon.com/cloudformation/home) to verify whether the bucket was provisioned successfully. ### Set S3 Bucket as Upload Location By default, your project/folder uses Synapse's default S3 storage location. You can use the external bucket configured above via the web or programmatic clients. **Web** Navigate to your project or folder of interest, then select**Tools** , and**Change Storage Location** . In the resulting pop-up, select the**Amazon S3 Bucket** option and fill in the relevant information, where **Bucket** is the name of your external bucket, the optional **Base Key** is the name of the folder in your bucket to upload to, and **Banner** is a short description such as who owns the storage location: ![external-s3.png](https://docs.synapse.org/__attachments/a_3f16b1b89df7e20e5308466a3ab975cebea3e83db87b18e2f805ea180d31aa38/external-s3.png?cb=38f6ea0f8ea3fbf1a4a2c716a1d48e03) **Python** Python # Set storage location import synapseclient import json syn = synapseclient.login() PROJECT = 'syn12345' destination = {'uploadType':'S3', 'concreteType':'org.sagebionetworks.repo.model.project.ExternalS3StorageLocationSetting', 'bucket':'nameofyourbucket'} destination = syn.restPOST('/storageLocation', body=json.dumps(destination)) project_destination ={'concreteType': 'org.sagebionetworks.repo.model.project.UploadDestinationListSetting', 'settingsType': 'upload'} project_destination['locations'] = [destination['storageLocationId']] project_destination['projectId'] = PROJECT project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination)) **R** R #set storage location library(synapser) library(rjson) synLogin() projectId <- 'syn12345' destination <- list(uploadType='S3', concreteType='org.sagebionetworks.repo.model.project.ExternalS3StorageLocationSetting', bucket='nameofyourbucket') destination <- synRestPOST('/storageLocation', body=toJSON(destination)) projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting', settingsType='upload') projectDestination$locations <- list(destination$storageLocationId) projectDestination$projectId <- projectId projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination)) ### Adding Files in your S3 Bucket to Synapse If your bucket is set for read-write access, files can be added to the bucket using the standard Synapse interface (web or programmatic). If the bucket is read-only or you already have content in the bucket, you will have to add representations of the files in Synapse programmatically. This is done using a `FileHandle`, which is a Synapse representation of the file. **Python** Python # create filehandle fileHandle = {'concreteType': 'org.sagebionetworks.repo.model.file.S3FileHandle', 'fileName' : 'nameOfFile.csv', 'contentSize' : "sizeInBytes", 'contentType' : 'text/csv', 'contentMd5' : 'md5', 'bucketName' : destination['bucket'], 'key' : 's3ObjectKey', 'storageLocationId': destination['storageLocationId']} fileHandle = syn.restPOST('/externalFileHandle/s3', json.dumps(fileHandle), endpoint=syn.fileHandleEndpoint) f = synapseclient.File(parentId=PROJECT, dataFileHandleId = fileHandle['id']) f = syn.store(f) #### R R # create filehandle fileHandle <- list(concreteType='org.sagebionetworks.repo.model.file.S3FileHandle', fileName = 'nameOfFile.csv', contentSize = 'sizeInBytes', contentType = 'text/csv', contentMd5 = 'md5', storageLocationId = destination$storageLocationId, bucketName = destination$bucket, key ='s3ObjectKey') fileHandle <- synRestPOST('/externalFileHandle/s3', body=toJSON(fileHandle), endpoint = 'https://file-prod.prod.sagebase.org/file/v1') f <- File(dataFileHandleId=fileHandle$id, parentId=projectId) f <- synStore(f) See the [REST docs](http://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/project/ExternalS3StorageLocationSetting.html) for more information on setting external storage location settings using our REST API. ## Setting up an External Google Cloud Storage Bucket Follow the documentation on Google Cloud's site to [**Create a Bucket**](https://cloud.google.com/storage/docs/creating-buckets). Make the following adjustments to customize it to work with Synapse: * Select the newly created bucket and click the **Permissions** tab. * Select the **Add members** button and enter the member `synapse-svc-prod@uplifted-crow-246820.iam.gserviceaccount.com`. This is Synapse's service account. Give the account the permissions **Storage Legacy Bucket Reader** and **Storage Object Viewer** for read permission. To allow Synapse to upload files, additionally grant the**Storage Legacy Bucket Writer** permission. To register the bucket with Synapse, you also need to create an object that proves to the Synapse service that you own this bucket. This can be done by creating a file named [**owner.txt**](https://docs.synapse.org/assets/downloads/owner.txt) that contains a **line or comma separated** list of *user identifiers* that are allowed to register the bucket and uploading it to your bucket. Valid *user identifiers* are: a Synapse user ID or the ID of a team that you are a member of. ![ownerTxt.png](https://docs.synapse.org/__attachments/a_794708d635ff3b315eba5066bf3983765cea995b8854eeb9d450ecd60a43c504/ownerTxt.png?cb=ea0d34317703d2b2ef6531b2602230c7) The ID of the user or the team can be obtained by navigating to the user profile or to the team page, the ID is the numeric value shown in the browser URL bar after the *Profile:* or *Team:* prefixes: ![browserUserId.png](https://docs.synapse.org/__attachments/a_4fbd3214228d50e96934cdbc65da51cf98e22e37b4d317ce6fd6611d48bb94d8/browserUserId.png?cb=ce72e9c2217411b064d09f3e3d86e79f) ![browserTeamId.png](https://docs.synapse.org/__attachments/a_569ce004eb2a534030b20c225f0acc268ff705b6de313d7c1d2ed2fee4886996/browserTeamId.png?cb=ea7f6e96e4e1f1e7c2282c95c1b5b0c1) You can upload the file with the Google Cloud Platform Console, or using the command line [gsutil application](https://cloud.google.com/storage/docs/gsutil). **Note:** Files in an external bucket will not be automatically added to Synapse. To add files to Synapse that are already in your bucket, see below. **Command line** # copy your owner.txt file to your s3 bucket gsutil cp owner.txt gs://nameofmybucket/nameofmyfolder **Web** Navigate to your bucket on the Google Cloud Console and select the **Upload files** button to upload your text file into the folder where you want your data. ### Enable Cross-Origin Resource Sharing (CORS) Follow the instructions for [Setting CORS on a bucket](https://cloud.google.com/storage/docs/configuring-cors). You may have to install the [gsutil application](https://cloud.google.com/storage/docs/gsutil). The configuration must include Synapse as a permitted `origin`. An example CORS configuration that would allow this is: [ { "maxAgeSeconds": 3000, "method": ["GET", "POST", "PUT", "HEAD"], "origin": ["*"], "responseHeader": ["Content-Type"] } ] Using **gsutil**, you can set the CORS configuration with the command: gsutil cors set cors-json-file.json gs://example-bucket where `cors-json-file.json` is a local file that contains a valid CORS configuration. For more information, please read: [Configuring cross-origin resource sharing (CORS)](https://cloud.google.com/storage/docs/configuring-cors). ### Set Google Cloud Bucket as Upload Location By default, your project uses the Synapse default storage location. You can use the external bucket configured above via our programmatic clients or web client. **Python** # Set storage location import synapseclient import json syn = synapseclient.login() PROJECT = 'syn12345' destination = {'uploadType':'GOOGLECLOUDSTORAGE', 'concreteType':'org.sagebionetworks.repo.model.project.ExternalGoogleCloudStorageLocationSetting', 'bucket':'nameofyourbucket', 'baseKey': 'nameOfSubfolderInBucket' # optional, only necessary if using a subfolder in your bucket } destination = syn.restPOST('/storageLocation', body=json.dumps(destination)) project_destination ={'concreteType': 'org.sagebionetworks.repo.model.project.UploadDestinationListSetting', 'settingsType': 'upload'} project_destination['locations'] = [destination['storageLocationId']] project_destination['projectId'] = PROJECT project_destination = syn.restPOST('/projectSettings', body = json.dumps(project_destination)) **R** #set storage location library(synapser) library(rjson) synLogin() projectId <- 'syn12345' destination <- list(uploadType='GOOGLECLOUDSTORAGE', concreteType='org.sagebionetworks.repo.model.project.ExternalGoogleCloudStorageLocationSetting', bucket='nameofyourbucket', baseKey='nameOfSubfolderInBucket' # optional, only necessary if using a subfolder in your bucket } ) destination <- synRestPOST('/storageLocation', body=toJSON(destination)) projectDestination <- list(concreteType='org.sagebionetworks.repo.model.project.UploadDestinationListSetting', settingsType='upload') projectDestination$locations <- list(destination$storageLocationId) projectDestination$projectId <- projectId projectDestination <- synRestPOST('/projectSettings', body=toJSON(projectDestination)) **Web** Navigate to your project or folder of interest, then select**Tools** , and**Change Storage Location** . In the resulting pop-up, select the **Google Cloud Storage Bucket** option and fill in the relevant information, where **Bucket** is the name of your external bucket, **Base Key** is the name of the folder in your bucket to upload to, and **Banner** is a short description such as who owns the storage location. ### Adding Files in Your Google Cloud Bucket to Synapse If your bucket is set for read-write access, files can be added to the bucket using the standard Synapse interface (web or programmatic). If the bucket is read-only or you already have content in the bucket, you will have to add representations of the files in Synapse programmatically. This is done using a `FileHandle`, which is a Synapse representation of the file. **Python** Python externalFileToAdd = 'googleCloudObjectKey' # put the key for the file to add here # create filehandle fileHandle = {'concreteType': 'org.sagebionetworks.repo.model.file.GoogleCloudFileHandle', 'fileName' : 'nameOfFile.csv', 'contentSize' : "sizeInBytes", 'contentType' : 'text/csv', 'contentMd5' : 'md5', 'bucketName' : destination['bucket'], 'key' : externalFileToAdd, 'storageLocationId': destination['storageLocationId']} fileHandle = syn.restPOST('/externalFileHandle/googleCloud', json.dumps(fileHandle), endpoint=syn.fileHandleEndpoint) f = synapseclient.File(parentId=PROJECT, dataFileHandleId = fileHandle['id']) f = syn.store(f) **R** externalFileToAdd <- 'googleCloudObjectKey' # put the key for the file to add here # create filehandle fileHandle <- list(concreteType='org.sagebionetworks.repo.model.file.GoogleCloudFileHandle', fileName = 'nameOfFile.csv', contentSize = 'sizeInBytes', contentType = 'text/csv', contentMd5 = 'md5', storageLocationId = destination$storageLocationId, bucketName = destination$bucket, key = externalFileToAdd) fileHandle <- synRestPOST('/externalFileHandle/googleCloud', body=toJSON(fileHandle), endpoint = 'https://file-prod.prod.sagebase.org/file/v1') f <- File(dataFileHandleId=fileHandle$id, parentId=projectId) f <- synStore(f) Please see the [REST docs](http://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/project/ExternalGoogleCloudStorageLocationSetting.html) for more information on setting external storage location settings using our REST API. **See also:** [Compute Directly on Data in Synapse or S3](https://docs.synapse.org/synapse-docs/compute-directly-on-data-in-synapse-or-s3.md) --- language: "en" --- # Data Access Types One of the ways that Synapse governance supports responsible data sharing is through data access types. While [user account types](https://docs.synapse.org/synapse-docs/synapse-user-account-types) are applied to the user, data access types are applied to the data itself. The parameters of data access for any dataset is defined by contributors of that data. Data access can be controlled in two layers: sharing settings and conditions for use. Sharing settings enable you to list the individuals or teams who can view your project, and the permissions those groups have with respect to a specific dataset. You can use sharing settings to authorize who can view, edit, download, or delete data. The Access \& Compliance Team (ACT) can add an additional layer of data protection by applying conditions for use\*. These restrictions define *how* users who have permission to download data may use it. Conditions for use may include IRB approval or other restrictions that you define as the data contributor. When both sharing settings and conditions for use are applied, then users must qualify for data access through *both*of these restrictions. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For information on sharing settings and conditions for use, including how to set them, see [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use). Sharing settings are included in the Synapse [++Basic Plan++](https://docs.synapse.org/synapse-docs/sage-offerings). \* Conditions for use is a service provided through a Self-Managed or Data Coordination [Synapse Plan](https://docs.synapse.org/synapse-docs/sage-offerings). *** ** * ** *** There are four data access types in Synapse: ## Private Access Data Data set to private access is visible only to you and other users whom you select in sharing settings. It is not viewable to any other Synapse users. This is the default setting when you create a new project in Synapse. ## Controlled Access Data Data set to controlled access can be available to registered, certified, and validated users that fulfil specific requirements for data access through conditions of use. Controlled access typically protects human data that contains sensitive information. Misuse of this data could potentially cause harm to those individuals or groups. Therefore, data contributors who choose a Self-Managed and Data Coordination [++Synapse Plan++](https://docs.synapse.org/synapse-docs/sage-offerings) can define conditions for use to specify how the data can be used by those who have access to it. Controlled access data can only be downloaded and used by those authorized to do so, and **this data may never be redistributed**. Any Synapse user wishing to use controlled access data must request access. Examples of controlled access data may include one or more of the following: * Data at risk of re-identifying research participants * Individual-level human "-omics" data * Data from "vulnerable" populations as defined using OHRP guidelines * Data generated with restrictions or requirements for use as outlined in informed consents or legal agreements To learn more about conditions for use and how to apply them to your data, see [Conditions for Use](https://docs.synapse.org/synapse-docs/synapse-governance#Synapse-Terms-and-Conditions-of-Use). ## Open Access Data Data set to open access is available to all registered Synapse users, without additional use limitations. Typically, open access data include: * Data from non-human organisms, species, or strains * Non-biological data, like data used for the calibration of instruments or methods * Human data that are: * Publicly available elsewhere * De-identified and non-sensitive, with no known sharing or use restrictions * Self-contributed and unambiguously consented for open data sharing and use ## Anonymous Access Data Data set to anonymous access is available for anyone (even anonymous, non-registered users) Anonymous access data could be: * Downloadable wiki content, such as newsletters or announcements of data releases * Metadata about a file or dataset --- language: "en" --- # Datasets A dataset is a collection of files that already exist in Synapse that may be hosted in one or more Synapse projects or folders. You can create a dataset that includes any files that you have read access to, whether you have own/edit access to them or not. You can use a dataset to: * Collect and distribute a set of files generated from the same study or project * Create a single item to represent a group of files that exist across disparate projects or folders The main use cases of datasets are to allow you to collect and share immutable sets of items, which: * You created and want to distribute to the community * You created and want to connect with a publication or tool * You found in Synapse and used as part of your own research, and want to distribute Although a dataset is similar to a [file view](https://docs.synapse.org/synapse-docs/views.md)++,++ it serves different purposes. While a file view allows you to set a scope for a folder that could be continuously updating, a dataset includes specific versions of files that you determine when setting it up. After creating a dataset, it will exist as a draft version, meaning you can continue editing it as you wish. You can also create a stable (static) version of the dataset at that point in time, which cannot be changed. You can share this stable version with others, or link it to a publication, by [minting a DOI](https://docs.synapse.org/synapse-docs/digital-object-identifiers-dois.md). Since you can create and share a dataset with files you do not own, it's important to ensure that you follow the [Synapse Terms and Conditions of Use](https://s3.amazonaws.com/static.synapse.org/governance/SageBionetworksSynapseTermsandConditionsofUse.pdf?v=5). ## When to use a dataset vs. a file view As mentioned, a dataset is similar to a [file view](https://docs.synapse.org/synapse-docs/views.md) in that its purpose is to group a specific set of files together, there are distinct differences between the two, which may determine when you would use one over the other. Review the table below for a summary of these differences. | | **Datasets** | **File Views** | |-----------------------------------|--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| | Underlying object type | [Dataset](https://docs.synapse.org/rest/org/sagebionetworks/repo/model/table/Dataset.html) | [EntityView](https://docs.synapse.org/rest/org/sagebionetworks/repo/model/table/EntityView.html) | | Method for adding files | Select specific versions of individual files | Select projects and folders, including the latest version of all contained files, tables, or datasets | | View and query file annotations | ✔ | ✔ | | Edit file annotations in bulk | | ✔ | | Versioning/snapshot functionality | ✔ | ✔ | | DOI functionality | ✔ | ✔ | | Limit on number of files | 10,000 | up to 350,000,000 with appropriate project/folder structure (see [View Limits](https://help.synapse.org/docs/Views.2011070739.html#Views-ViewLimits)) | ## How to create a dataset ### Create a dataset 1. Navigate to the project that you want to create the dataset in 2. Click the **Datasets**tab 3. Click **Add Dataset** 4. In the **Create Dataset**window, enter a name for the dataset 5. Click **Finish** You will now be directed to the new dataset that you just created (it will be empty at this point). Notice the banner indicating that this is a draft version of the dataset. A draft dataset should not be distributed externally until it is finalized by creating a stable version. See the section [Create a Stable Version](https://help.synapse.org/docs/Datasets.2611281979.html#Datasets-Createastableversion) below for more information. ### Add files to the dataset 1. In the dataset, click **Add Items** 2. In the **Add Files to Dataset** window, browse for the file(s) you want to add 1. Click on the name of a project to see all folders, files, and tables contained within that project. Note that only files can be selected and added to the dataset 2. If you want to see the contents of an individual folders, click the dropdown arrow next to a project name, or next to a folder, to reveal all of its contents. This will allow you to select in individual files contained within 3. You can also search for individual files using the **Search for Files** tool (Note that you cannot use this tool to search for folders or projects, only individual files) 3. Click the checkbox next to any of the file(s) that you want to add. If you want to add all files from within a folder, you can click the general checkbox at the top of the list to add all contents. You can also select which version of the file you want to appear in your dataset. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information at [Versioning Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/versioning-tables-views-and-datasets.md). Your selections will appear in the **Selected**box at the bottom. They will remain in this "selected" status, even as you navigate through other folders and files. You can remove individual selections from here if necessary. 4. Once you have selected all of the files you want, click **Add Files** . All files from your **Selected**box will be added to the dataset. At this point, before saving the dataset, you can still add or remove files from the dataset, or change the version of any files (see screenshot below) 5. Click **Save**to save your current selection and return to the draft dataset In the screenshot below, notice how there are several actions you can take after adding new files, before saving this version. You can use the checkmarks to select any of the files and remove them, or use the **Version** dropdowns to change the version selection of any file. You can also add new files. Note that you can also edit the draft dataset after saving, but if you create a stable version then it will reflect the selection that you included at the time of creation. See the section [Create a Stable Version](https://help.synapse.org/docs/Datasets.2611281979.html#Datasets-Createastableversion) below for more information. ![Screen Shot 2022-03-18 at 1.04.11 PM.jpg](https://docs.synapse.org/__attachments/a_825fd0940348bcfad19663014cdb44345aa8e8676677b6de9fe7328866c2f4f9/Screen%20Shot%202022-03-18%20at%201.04.11%20PM.jpg?cb=e17fe0aaa478c8e64e2c1db9ee9680a5) ## How to use a dataset Once you have created a draft dataset, there are a number of things you can do to it, similar to other features in Synapse. This includes: * Create a stable version (a static snapshot of the dataset) * Edit sharing settings * Annotate the dataset with metadata (in order to query for sets of datasets) * Create a wiki (add documentation of the dataset using the wiki) * Edit the dataset column schema * Mint a DOI These actions are described below or linked to other help articles. ### Create a stable version A dataset can exist as a draft or stable version. A draft dataset is mutable, meaning that it can be edited. A stable version is a snapshot of the dataset at the moment the version was created. The version will have a synID which is appended with a number based on which version it is. For example, syn123456.2 would be version two of syn123456. Only stable versions should be shared with others, or included in downstream resources, as only stable versions are immutable (static and uneditable). If a file is deleted from Synapse, its metadata will still be visible in any stable dataset version that included that file. However, if another user clicks on that dataset version, they will find that it no longer exists. Such a deleted file may still be visible, but it no longer physically exists. It is important to note that the wiki, sharing settings, and annotations remain the same between the draft dataset and the stable version. Here's how to create a stable version: 1. Click **Dataset Tools** and select **Create a Stable Dataset Version** from the dropdown menu 2. In the **Create Stable Version**window, add an appropriate label for the version, and a comment if necessary. Note that you do not need to add a version number, since is is already added for you You will now see the new version, as well as the full version history. From here, you can go back to your draft. ### Edit sharing settings In the dataset, click on **Dataset Tools** , and select **Dataset Sharing Settings**from the dropdown menu. This will show you the current sharing settings of the dataset. Note that sharing settings of the dataset will be inherited from any parent projects or folders. If you want to have different settings on a specific file, you can create local sharing settings and then modify them. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on sharing settings, see [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). ### Annotate the dataset with metadata If your dataset is included in a view, you may wish to customize how you or others are able to query your dataset. If this is the case, you can add annotations so that you and other users can query for this dataset on custom keys. Another reason for adding annotations is to make your dataset findable using the search tool in Synapse. To add annotations to a dataset: 1. Once in the dataset, click **Dataset Tools** and select **Annotations**from the dropdown list 2. In the **My Dataset** window, click **Edit** 3. As the on-screen instructions state, click the **Add**icon to begin adding annotations 4. Complete your annotations using the fields provided * Use the **+** button to the right of each row to add a new value for any **Key** * Use the **x**button to the right of each row to delete that row * Click the **Add**icon to add another Key 5. Click **Save** ### Edit the dataset schema You can customize the visible columns of a dataset. These columns will be auto-populated based on the annotation values of the underlying files in the dataset. Here's how to customize these columns: 1. In the dataset, click **Dataset Tools** , and select **Show Dataset Schema**from the dropdown menu 2. Click **Edit Schema**at the bottom of the table 3. In the resulting **Edit Columns**window, you can add columns to your dataset schema using any combination of these three options: * Click **Add Column**to manually add individual columns one by one. If these columns exist as annotations on one or more of the files in the Dataset, the values will be displayed in the Dataset. You cannot use a Dataset to bulk annotate files, so do not add columns that do not already exist as annotations, since this will not serve any purpose. * Click **Add Default Dataset Columns**to add the default columns used in Datasets * You can then customize this list by removing any of the default columns you don't want to be included---to do so, click the checkbox next to any column(s), followed by the trash can icon at the top. * Click **Import columns**to import columns from another table in Synapse. Again, only columns which already exist as underlying annotations will be relevant. 4. Once you have added all columns of interest, you can: * Use the arrows at the top to reorder the columns * Enter any values as needed in the **Restrict Values**column * Select/change any column facet using the **Facet**dropdown 5. Click **Save** ### Mint a DOI (digital object identifier) You can use a DOI (Digital Object Identifier) to generate a permanent link to the dataset. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information and instructions at [Digital Object Identifiers (DOIs)](https://help.synapse.org/docs/Digital-Object-Identifiers-(DOIs).1972405096.html). --- language: "en" --- # Digital Object Identifiers (DOIs) A [Digital Object Identifier](https://www.doi.org/) (DOI) is a persistent identifier assigned to uniquely identify a digital object. A DOI is defined by a digital location like a URL and a description of the object. This description includes attribution and a creation or publication date. Creating and assigning a new DOI is commonly known as "minting". Minting a DOI for things in Synapse allows you to reference them when used elsewhere, such as in a publication or on an external website. DOIs are available in Synapse for projects, files, folders, tables, and views. DOIs that are for objects stored in Synapse have a prefix of `doi:10.7303`. They are then followed by the Synapse ID of the object being linked to, such as `syn2580853`, which is a Synapse project. The DOI `doi:10.7303/syn2580853` can be represented as a URL, , which automatically redirects to the associated Synapse project, the [AMP-AD Knowledge Portal](https://www.synapse.org/#!Synapse:syn2580853). DOIs can be minted for specific versions of files, tables, and views, which ensures that other researchers access the same data that you published, even if it's updated later. When minting a DOI for a version, the synID and URL will have ".xx" appended to it, where "xx" indicates the version number (for example, `10.7303/syn3539963.2`). If a new version is created, a new DOI will also need to be created to reference the new version. If a DOI is minted without a version number, then the DOI will always link to the latest version of the object. This can be useful when referencing a living, changing document, where consistent access to the same set of data is less crucial. While DOIs are designed to be public, minting a DOI does not change the [sharing settings](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md) on an object. However, the metadata minted with a DOI, such as title and authors, will be retrievable from other sources, such as [CrossCite](https://citation.crosscite.org/), even if the object is private in Synapse. ## Minting DOIs 1. Navigate to the data or project you'd like to create a DOI for. 2. From the **Tools Menu** , select the option for **Create DOI** for the object in question, or choose **Create DOI** from the **Project Settings** menu if minting a DOI for an entire project. 3. Fill out the form as needed. You can add other creators if there are other individuals who contributed to your work. You can select a resource type which describes what type of object it is. The title and publication year can be changed, but this is not advisable. If you do not see these options, you likely do not have permission to create a DOI for the object. Only users with Edit access or above may mint DOIs for objects. You can also update an existing DOI if information has changed. --- language: "en" --- # Discussion Forums Discussion forums are a space to communicate with others, similar to a message board. The discussion forum is visible to users who have access to the project. ## Creating a New Message Thread The **Discussion** tab can be found on the project main page. Anyone with project access may view project discussion threads, create new threads, or reply to existing threads. ![Discussion_Main.png](https://docs.synapse.org/__attachments/a_66850da0ea8c7498e39c36ee7c0ac6aaa6414bc387bb8abda7b6551f11dcc4b4/Discussion_Main.png?cb=3e8c668842c11a388421af0d7dbabd64) To create a new thread, click the **New Thread** button at the top of the Discussion tab. ![NewThread_Dialogue.png](https://docs.synapse.org/__attachments/a_63697b06135deb9419c95f97f566ca46f1ee013f76d9b7fc7517f197921f0ed4/NewThread_Dialogue.png?cb=f0df1db6691bf0f1598372d9cddfabcf) This opens a **New Thread** dialog box where you can edit text, insert pictures, code, TeX, widgets, or Markdown script. As with wiki editing, a formatting guide is available for Markdown, TeX, and code block examples. **Tip:** You may tag people by including an '@' symbol followed by a username. Tagging users in discussion threads will send them a notification, so use this feature to invite others into the discussion. Tagged users will only receive a notification if they were tagged in the initial thread creation. Tagging users after starting a thread will not send a notification. ## Managing Notifications You may subscribe to a discussion forum for a particular project by clicking the **Follow** button on the main discussion page. Once you've subscribed to a forum, you will receive an email to your Synapse-registered email address whenever a new thread is created. You can use the **Unfollow** button at any time to discontinue this subscription. To receive email notifications for replies to individual threads, click the **eye** icon, which is visible from within the thread of interest. If you reply to a thread, you begin following the thread automatically. ![UnfollowThread.png](https://docs.synapse.org/__attachments/a_3490153d99c0428578d73ceed1e878e6a3d04fe69eeb34e23a1853b90eadb613/UnfollowThread.png?cb=3c4300c08408fb6820ec995b88468e51) ## Moderating a Discussion Forum ### Deleting and Restoring Threads Users with administrative privileges on a project also have moderator privileges for that project's forum. Moderators may delete threads and individual replies from all users using the **trash can** icon next to the original post. In case of accidental deletion, moderators may also undelete threads to restore them. To view deleted threads, click on **Discussion Tools** and **Show Deleted Threads** to see a list of threads that are currently in the trash can. Once you've found the thread you'd like to restore, click the thread and use the **Discussion Tools** menu to select **Restore Thread**. ### Pinning Threads Moderators may emphasize important threads by selecting the **pin** icon to bring the post to the top of the forum. The thread will remain at the top of the list even after newer threads are posted. Moderators may also unpin threads. --- language: "en" --- # Downloading Data From the Synapse UI Remember, when downloading this way, the maximum size of a download is 5 GB, or a maximum of 100 files if using the download cart. Once you have [found and gained access to your data of interest](https://docs.synapse.org/synapse-docs/finding-and-downloading-data.md), here's how to download that data. Reference the screenshots below for a visual representation of the instructions. Within the project, you may see a series of folders. Any standalone files are downloadable using the down arrow icon in the **Download** column for that file **(1)**. You can click the \> arrow next to any folder to expand it and view its files within **(2)**. To download any individual file from here, click the download icon in the **Download** column for that file **(1)**, which will add that file to your download cart. ![downloading2.png](https://docs.synapse.org/__attachments/a_7cf12b6859bce787a5a4fb59ff36d11706ba4040aba7fb13f4b244ed3b40ec22/downloading2.png?cb=b5cc41cb2674b00015f9bb46926163cf) Alternatively, if you want to download every file within a folder, you can do so more efficiently. Click on the name of the folder (instead of just expanding it). On this new page with just that folder and its contents, click **Download Options** **(3)** followed by **Add to Download Cart** **(4)**. You'll be notified of the number and size of files, and asked if you wish to proceed---click **Add** **(5)**. ![downloading3.png](https://docs.synapse.org/__attachments/a_523ed6eafeceb7cc3847f1d3b92d875be1f429f4d07be3610a956728780d5c1d/downloading3.png?cb=f00281b49c900b131d30dbd451f15a6a) Repeat this process for as many files (individual and within folders) that you want to download! As you add items to your download cart, notice that this gets reflected in the **Downloads** icon of your Synapse toolbar on the left **(6)**. Click this icon once you are ready to download all files in your cart (at this point, they are not downloaded to your computer yet). In your download cart, review all the items in the list. From here, you can use the **Action** column to remove any files that you no longer wish to download **(7)**. When you're ready to download, click **Download As .Zip Packages** **(8)**. This will reveal a **Create Your Download Package** box below, which will prompt you to enter a package name **(9)**. Enter a name that will be easy to find and follows protocols for your project. Then, click **Download Package** **(10)**. The zipped package will now be available on your computer. ![downloading4.png](https://docs.synapse.org/__attachments/a_4a300247b23436c5be506e530b032aca65eca56e71eed770e60dd312b67b3631/downloading4.png?cb=2184e5c32487cdb6ece51c62fac76ae1) --- language: "en" --- # Downloading Data Programmatically Data in Synapse can be downloaded using our programmatic clients (Python, R, and command line), or [from the Synapse UI](https://docs.synapse.org/synapse-docs/downloading-data-from-the-synapse-ui.md). In this guide, you will learn the basic commands to download data programmatically. ## Downloading Files Before you begin, it is important to understand that most items in Synapse have a unique identifier associated with them. This identifier is called a Synapse ID, or a synID. The synID format is the prefix "syn" followed by 8 numbers (for example, syn12345678). Items that have unique synIDs in Synapse are: files, folders, projects, tables, views, wikis, links, and Docker repositories. You can use synIDs to refer to specific items when working with Synapse programmatically. When using the Python, R, or command line clients, files can be downloaded by using the `get` command. Downloaded files are stored and/or registered in a cache. By default, the cache location is in your home directory in a hidden folder named `.synapseCache`. Whenever the `get` function is invoked, the cache is checked to see if the same file is already present by checking its MD5 checksum. If it already exists, the file will not be downloaded again. In other words, if the current version of a file has already been downloaded, Synapse will not re-download the same file. For the Python and R clients, the default download location is the Synapse cache. The command line client downloads to your current working directory. On the web, your own browser settings determine the download location for files. The Synapse cache is not updated to reflect downloads through a web browser. In all cases you can specify the directory in which to download the file. For example, to download the experimental protocol on [Adult Mouse Cardiac Myocyte Isolation](https://www.synapse.org/#!Synapse:syn3158111) (`syn315811`) from the [Progenitor Cell Biology Consortium (PCBC)](https://www.synapse.org/#!Synapse:syn1773109) you would run the following: **Command line** synapse get syn3158111 **Python** import synapseclient syn = synapseclient.Synapse() syn.login() entity = syn.get("syn3158111") **R** library(synapser) synLogin() entity <- synGet("syn3158111") Once a file has been downloaded, you can find the file path using the following: **Command line** # When downloading using the command line client, it will print the filepath of where the file was saved to. **Python** filepath = entity.path **R** filepath <- entity$path ## Downloading a Specific File Version If there are multiple versions of a file, a specific version can be downloaded by passing the `version` parameter. In this example, there are multiple versions of an [miRNA FASTQ file](https://www.synapse.org/#!Synapse:syn3260973) (`syn3260973`) from the Progenitor Cell Biology Consortium. To download the first version: **Command line** synapse get syn3260973 -v 1 **Python** entity = syn.get("syn3260973", version=1) **R** entity <- synGet("syn3260973", version=1) See [Versioning](https://docs.synapse.org/synapse-docs/versioning.md) for more details. ## Downloading Linked Data When you click on a link on the Synapse website, it will redirect you to the linked entity. The `followLink` parameter will have to be specified when using the programmatic clients or you will only retrieve the link itself without downloading the linked entity. **Command line** synapse get syn1234 --followLink **Python** import synapseclient syn = synapseclient.login() linkEnt = syn.get("syn1234") entity = syn.get("syn1234", followLink=True) **R** library(synapser) synLogin() linkEnt = synGet("syn1234") entity = synGet("syn1234", followLink=TRUE) ## Downloading Location To override the default download location, you can specify the `downloadLocation` parameter. **Command line** synapse get syn00123 --downloadLocation /path/to/folder **Python** entity = syn.get("syn00123", downloadLocation="/path/to/folder") **R** entity <- synGet("syn00123", downloadLocation="/path/to/folder") ## Finding and Downloading Files via Annotations Files can be [annotated](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md) in Synapse to help organize your data and make files findable. In order to search the annotations, you must create a [file view](https://docs.synapse.org/synapse-docs/views.md) first. For example, the [PCBC Project](https://www.synapse.org/#!Synapse:syn1773109) has a [table](https://www.synapse.org/#!Synapse:syn7511263) listing sequencing data files that are annotated. To find all mRNA fastq files originating from CD34+ cells in the we can query by: **Command line** synapse query 'select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"' **Python** results = syn.tableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"') **R** results <- synTableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"') df <- as.data.frame(results) Once you've queried for the files of interest, they can be downloaded using the following: **Command line** synapse get -q 'select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"' **Python** results = syn.tableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"') entity = [syn.get(r['file.id']) for r in results] **R** results <- synTableQuery('select * from syn7511263 where dataType="mRNA" AND fileType="fastq" AND Cell_Type_of_Origin="CD34+ cells"') df <- as.data.frame(results) entity <- lapply(df$file.id, function(x) synGet(x)) ## Recursive Downloading The folder structure that is present on Synapse can be maintained by recursive downloading. **Command line** synapse get -r syn2390898 **Python** import synapseutils import synapseclient syn = synapseclient.login() files = synapseutils.syncFromSynapse(syn, 'syn2390898') **R** # Unfortunately, this feature is not available in the R client ## Downloading Wikis The structure of a wiki page can be extracted through the R and Python clients. The ID, title, and parent wiki page of each sub-wiki page is also determined through the same method. **Python** wiki = syn.getWikiHeaders("syn00123") **R** entity <- synGet("syn00123") wiki <- synGetWikiHeaders(entity) The Markdown content within a wiki page can be downloaded if you know the synID and page ID for the wiki. The wiki page ID can either be obtained through the above method or can be found in the URL. For example, in the URL `www.synapse.org/#!Synapse:syn00123/wiki/123456`, the last 6 digits of the URL path is the wiki page ID (123456). **Python** wiki = syn.getWiki("syn00123", 12345) **R** entity <- synGet("syn00123") wiki <- synGetWiki(entity, 12345) ## Downloading in Bulk Files can be downloaded in bulk using the `syncFromSynapse` function found in the [synapseutils](https://python-docs.synapse.org/reference/synapse_utils/#synapseutils.sync.syncFromSynapse) helper package. This function crawls all the subfolders of the project or folder that you specify and retrieves all the files that have not been downloaded. By default, the files will be downloaded into your `synapseCache`, but a different download location can be specified with the `path` parameter. If you do download to a location out side of `synapseCache`, this function will also create a tab-delimited manifest of all the files along with their metadata (path, provenance, annotations, etc). **Python** # Load required libraries import synapseclient import synapseutils # login to Synapse syn = synapseclient.login(email='me@example.com', password='secret', rememberMe=True) # download all the files in folder syn123 to a local folder called "myFolder" all_files = synapseutils.syncFromSynapse(syn, entity='syn123', path='/path/to/myFolder') **R** # Load required libraries library(synapser) library(synapserutils) # login to Synapse synLogin(email='me@example.com', password='secret', rememberMe=TRUE) # download all the files in folder syn123 to a local folder called "myFolder" all_files = syncFromSynapse(entity='syn123', path='/path/to/myFolder') --- language: "en" --- # Downloading Data Programmatically From a Portal Portals are websites created to promote data sharing within a specific research community. These websites aggregate relevant data from Synapse, and they allow users to explore data, projects, people, and organizations within their research community. Sage Bionetworks hosts a variety of web portals for different research communities. The [AD Knowledge Portal](https://adknowledgeportal.synapse.org/), the [NF Data Portal](https://nf.synapse.org/), and the [PsychENCODE Knowledge Portal](https://psychencode.synapse.org/) are a few examples. The following guide describes how to programmatically download data discovered on a portal. ## Find Files Using Explore All entities in Synapse are automatically assigned a globally unique identifier used for reference with the format `syn12345678`. Often abbreviated to "synID", the ID of an object never changes, even if the name does. You will use a synID to locate the files you wish to download. Search the available data files via the **Explore** tab in the navigation bar. The**Explore** section presents several ways to select data files of interest. The top of the page displays pie charts that summarize the data files based on file annotations of interest, including **Assay** and **Tissue** , among others. Selection of one of these chart segments will filter the table below to subset the set of files. Alternatively, access the filters using the facet selection boxes to the left of the table. For this example, you will [download the *processed* data and *metadata* from the *MC-CAA* study in the Alzheimer's Disease (AD) Knowledge Portal](https://adknowledgeportal.synapse.org/Explore/Data?QueryWrapper0=%7B%22sql%22%3A%22SELECT%20*%20FROM%20syn11346063%22%2C%22limit%22%3A25%2C%22offset%22%3A0%2C%22selectedFacets%22%3A%5B%7B%22concreteType%22%3A%22org.sagebionetworks.repo.model.table.FacetColumnValuesRequest%22%2C%22columnName%22%3A%22study%22%2C%22facetValues%22%3A%5B%22MC-CAA%22%5D%7D%2C%7B%22concreteType%22%3A%22org.sagebionetworks.repo.model.table.FacetColumnValuesRequest%22%2C%22columnName%22%3A%22dataSubtype%22%2C%22facetValues%22%3A%5B%22processed%22%2C%22metadata%22%5D%7D%5D%7D). ## Download Files ### Command Line The Synapse command line client can be used to download all data and file annotations with a single command. The command line client is installed with the Synapse Python client, therefore [Python 3](https://www.python.org/downloads/) is required to [install the Synapse command line client](https://python-docs.synapse.org/tutorials/installation/). [Login](https://python-docs.synapse.org/tutorials/command_line_client/#login) to Synapse. If working on your personal computer, you may store your credentials locally by including the `--rememberMe` argument to allow automatic authentication with future Synapse interactions. This is recommended to prevent a case where you might accidentally share your password while sharing analytical code. To login with your username and password, e.g. synapse login -u -p --rememberMe A Synapse [access token](https://docs.synapse.org/synapse-docs/managing-your-account.md#Personal-Access-Tokens) is more secure than your password and is highly recommended to be used to login instead of using your password. synapse login -p --rememberMe From**Explore Data** in the portal, select the **Download Options** icon and **Programmatic Options** to visualize the command to download the data subset. ![programmatic-options-viz.png](https://docs.synapse.org/__attachments/a_32f2356274e0e1bf28d515a26164317ec40ef0d70ac3e86dc457a06ef5c99166/programmatic-options-viz.png?cb=323b1ed26d244e898baf3103395a9395) The command `synapse get` with the `-q` argument downloads files that meet the specified condition. In this example, all *processed* and *metadata* files from the *MC-CAA* study will be downloaded. Execute the following command from the directory where you would like to store the files. synapse get -q "SELECT * FROM syn11346063 WHERE ( ( "study" = 'MC-CAA' ) AND ( "dataSubtype" = 'processed' OR "dataSubtype" = 'metadata' ) )" Also in your working directory, you will find a `SYNAPSE_TABLE_QUERY_###.csv` file that lists the annotations associated with each downloaded file. Here, you will find helpful experimental details relevant to how the data was processed. Additionally, you will find important details about the file itself including the file version number. ### R In order to download data programmatically with R, you need a list of synIDs that correspond to the files. For downloading a large set of files, we recommend using the Synapse Python client. The Python client has been optimized for multi-threaded download and will provide you with faster download speeds. Once you have identified the files you want to download from **Explore Data** , use the **Export Table** option from **Download Options**. The table includes annotations associated with each downloaded file. ![export-table-viz.png](https://docs.synapse.org/__attachments/a_9399750ea4a219462a9f89bc163cefe6256242e39106994cee4a8b3a7bc4139a/export-table-viz.png?cb=840927800361c79bafe3982653b92882) You may choose to download the file as a `.csv` or `.tsv`. Files are named `Job-#### `(where # is a long set of numbers). Move this file to your working directory to proceed with the following steps. [Install the Synapse R client](https://r-docs.synapse.org/#installation) `synapser` to download data from Synapse. [Login](https://r-docs.synapse.org/articles/manageSynapseCredentials.html#manage-synapse-credentials) using your password or preferably an [access token](https://docs.synapse.org/synapse-docs/managing-your-account.md#Personal-Access-Tokens). R library(synapser) # using password synLogin("my_username", "password") # OR using an access token synLogin(authToken="token") Read the exported table into R replacing `Job-####` with the complete filename of the downloaded table. Create a directory to store files and download data using `synGet`. If `downloadLocation` is not specified, the files are downloaded to a hidden directory called `~/.synapseCache`. R exported_table <- read.csv("Job-####.csv") dir.create("files") lapply(exported_table$id, synGet, downloadLocation = "./files") The annotations in `exported_table` include experimental details relevant to how the data was processed. ### Python In order to download data programmatically, you need a list of synIDs that correspond to the files. Once you have identified the files you want to download from **Explore Data** , use the **Export Table** option from **Download Options**. The table includes annotations associated with each downloaded file. ![export-table-viz.png](https://docs.synapse.org/__attachments/a_9399750ea4a219462a9f89bc163cefe6256242e39106994cee4a8b3a7bc4139a/export-table-viz.png?cb=840927800361c79bafe3982653b92882) You may choose to download the file as a `.csv` or `.tsv`. Files are named `Job-####`, where # includes a long set of numbers. Move this file to your working directory to proceed with the following steps. [Install the Synapse Python client](https://python-docs.synapse.org/tutorials/installation/) `synapseclient` to download data from Synapse, the `pandas` library to read a csv file and the `os` module to make a directory. [Login](https://python-docs.synapse.org/tutorials/python_client/#authentication) to Synapse using your password or preferably an [access token](https://docs.synapse.org/synapse-docs/managing-your-account.md#Personal-Access-Tokens). Python import synapseclient, pandas, os syn = synapseclient.Synapse() # login using a password syn.login('my_username', 'password') # OR using an access token syn.login(authToken='token') Read the exported table into Python replacing `Job-####` with the complete filename of the downloaded table. Create a directory to store files and download data using `syn.get`. If `downloadLocation` is not specified, the files are downloaded to a hidden directory called `~/.synapseCache`. Python exported_table = pandas.read_csv("Job-####.csv") os.mkdir("files") [syn.get(x, downloadLocation = "./files") for x in exported_table.id] The annotations in `exported_table` include experimental details relevant to how the data was processed. *** ** * ** *** **Need More Help?** Ask a question in the Synapse [Help Forum](https://www.synapse.org/#!SynapseForum:default). Your feedback is key to improving our documentation, so [contact us](mailto:synapseinfo@sagebase.org) if something is unclear or open an [issue](https://sagebionetworks.jira.com/secure/CreateIssue.jspa?issuetype=3&pid=12124). --- language: "en" --- # Evaluating Submissions Synapse collects all submissions via Evaluation Queues, and you can view and monitor them using Submission Views. This tutorial will walk you through how to view the submissions, download them for evaluation, and upload their scores back to Synapse. Looking to automate this process? Check out the last section for some available resources! Need more help with infrastructure setup? Reach out to the [Challenges \& Benchmarking Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/18) and a CNB team member will be in touch. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about [Evaluation Queues](https://help.synapse.org/docs/Evaluation-Queues.1985151345.html). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about setting up Submission Views at [Creating the Submission View](https://help.synapse.org/docs/Evaluation-Queues.1985151345.html#EvaluationQueues-CreatingtheSubmissionView). *** ** * ** *** ## Retrieving Submissions Before you can retrieve submissions from Synapse, ensure you have at least "Can score" permissions for the relevant Evaluation Queue(s), otherwise, an error about permissions will be encountered. If you had created the Queues, you should already have "Admin" privileges, which grant you the necessary permissions to view and retrieve submissions. ### View Submissions Directly from Evaluation Queues To view submissions programmatically, you can use `syn.getSubmissions(EVALUATION_ID)` \[[ref](https://python-docs.synapse.org/en/stable/reference/client/#synapseclient.Synapse.getSubmissions)\]. For example: **Python** Python import synapseclient syn = synapseclient.login() evaluation_id = "9615516" submissions = syn.getSubmissions(evaluation_id) # returns a generator of submissions for submission in submissions: print(submission) # To view the submissions in a dataframe, use pandas. import pandas as pd submissions_df = pd.DataFrame(syn.getSubmissions(evaluation_id)) print(submissions_df) **R** R library(dplyr) library(jsonlite) library(synapser) synLogin() submissions <- synGetSubmissions("9615516")$asList() # To view the submissions in a dataframe submissions_df <- lapply(submissions, function(s) { data.frame( id = s$id, userId = s$userId, evaluationId = s$evaluationId, entityId = s$entityId, entityBundleJSON = s$entityBundleJSON, versionNumber = s$versionNumber, name = s$name, createdOn = as.character(s$createdOn), contributors = as.character(toJSON(s$contributors, auto_unbox = TRUE)), stringsAsFactors = FALSE ) }) %>% bind_rows() print(submissions_df) All new submissions are denoted with a status of `RECEIVED`. To view only new submissions, add `status="RECEIVED"` to the method call: **Python** Python submissions = syn.getSubmissions(evaluation_id, status="RECEIVED") **R** R submissions <- synGetSubmissions("9615516", status="RECEIVED")$asList() If you do not know the evaluation ID, you can first get the Evaluation object by using `syn.getEvaluationByName(EVALUATION_NAME)`and then pass it into `syn.getSubmissions()`. For example: **Python** Python evaluation = syn.getEvaluationByName("MY_EVALUATION_QUEUE_NAME") submissions = syn.getSubmissions(evaluation) **R** R evaluation <- synGetEvaluationByName("MY_EVALUATION_QUEUE_NAME") submissions <- synGetSubmissions(evaluation) If you also don't know the name, contact the Synapse user who created the Challenge project. They are likely the current admin of the Evaluation Queue(s) and can provide you with the evaluation ID(s). Otherwise, submit a support ticket to the [Challenges \& Benchmarking Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/18/group/27/create/203) for further assistance. ### View Submissions via Submission View If you have already set up a Submission View, you can view submissions by querying that table programmatically using `syn.tableQuery(QUERY)` \[[ref](https://python-docs.synapse.org/en/stable/reference/client/?h=tablequery#synapseclient.Synapse.tableQuery)\]: **Python** Python import pandas as pd import synapseclient syn = synapseclient.login() # View all submissions. view_id = "syn53293336" submissions_df = ( syn.tableQuery(f"SELECT * FROM {view_id}") .asDataFrame() .fillna("") ) print(submissions_df) **R** R library(synapser) library(tidyverse) synLogin() # View all submissions. view_id <- "syn53293336" submissions_df <- synTableQuery(str_glue("SELECT * FROM {view_id}")) %>% .$asDataFrame() %>% mutate(across(everything(), ~ ifelse(is.na(.) | . == "NaN", "", .))) ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) See [Using Advanced Search Queries](https://help.synapse.org/docs/Querying-Tables,-Views,-and-Datasets.2667642897.html#QueryingTables,Views,andDatasets-UsingAdvancedSearchQueries) for more examples of SQL-like queries supported by Synapse. To view only new submissions, add `status = 'RECEIVED'` as a clause to the query: **Python** Python import pandas as pd import synapseclient syn = synapseclient.login() # View only new submissions. view_id = "syn53293336" submissions_df = ( syn.tableQuery(f"SELECT * FROM {view_id} WHERE status = 'RECEIVED'") # add a clause .asDataFrame() .fillna("") ) print(submissions_df) **R** R library(synapser) library(tidyverse) synLogin() # View only new submissions. view_id <- "syn53293336" submissions_df <- synTableQuery(str_glue("SELECT * FROM {view_id} WHERE status = 'RECEIVED'")) %>% # add a clause .$asDataFrame() %>% mutate(across(everything(), ~ ifelse(is.na(.) | . == "NaN", "", .))) ### Download Submissions While there is not currently a feature on the Synapse web UI to download submissions, you can do so programmatically. Technically, you will be downloading *Submission objects*, which are copies of the entities submitted to an Evaluation Queue. As long as you have at least "Can score" permissions on the queue, you will be able to access and download these Submission objects. #### File Submissions You can directly download file submissions by using the submission ID with `syn.getSubmission(SUBMISSION_ID)` \[[ref](https://python-docs.synapse.org/en/stable/reference/client/?h=getsubmission#synapseclient.Synapse.getSubmission)\]. `syn.getSubmission()` is not the same as `syn.getSubmissions()`! `syn.getSubmission()` (no "s" at the end) retrieves the metadata for a single submission *and* downloads it if it's a file. Whereas `syn.getSubmissions()` only retrieves metadata of multiple submissions submitted to an Evaluation Queue. **Python** Python import synapseclient syn = synapseclient.login() submission_id = 9743445 syn.getSubmission(submission_id) # By default, all files are downloaded to ~/.synapseCache. To specify # a different location, use `downloadLocation`. syn.getSubmission(submission_id, downloadLocation="path/to/download") **R** R library(synapser) synLogin() submission_id <- 9743445 synGetSubmission(submission_id) # By default, all files are downloaded to ~/.synapseCache. To specify # a different location, use `downloadLocation`. synGetSubmission(submission_id, downloadLocation = "path/to/download") #### Docker Submissions For ++Docker++ submissions, you will need to use Docker to retrieve the submitted images; using `syn.getSubmission()` will not pull the image onto your machine. To get the image name associated with a submission, combine the `dockerRepositoryName` and `dockerDigest` from the Submission object, separated by an `@` symbol, e.g.`{dockerRepositoryName}@{dockerDigest}`. **CLI** Bash docker pull DOCKER_REPOSITORY_NAME@DOCKER_DIGEST **Python** Python import docker import synapseclient syn = synapseclient.login() # Setup Docker client. client = docker.from_env() # Get submission metadata with syn.getSubmission(..., downloadFile=False). submission_id = 9753713 submission = syn.getSubmission(submission_id, downloadFile=False) repo_name = submission.get("dockerRepositoryName") digest = submission.get("dockerDigest") # Attempt to pull Docker submission, otherwise output error message. try: client.images.pull(f"{repo_name}@{digest}") except docker.errors.APIError: print(f"Something went wrong with pulling submission {submission_id}") 💡 This example uses the [Docker SDK for Python](https://docker-py.readthedocs.io/en/stable/) to programmatically pull the images. **R** R library(synapser) library(tidyverse) synLogin() # Login to Synapse Docker Registry. system("docker login docker.synapse.org") # Get submission metadata with syn.getSubmission(..., downloadFile=False). submission_id <- 9753713 submission <- synGetSubmission(submission_id) repo_name <- submission$dockerRepositoryName digest <- submission$dockerDigest # Attempt to pull Docker submission, otherwise output error message. if (nzchar(repo_name) && nzchar(digest)) { exit_code <- system(str_glue("docker pull {repo_name}@{digest}")) if (exit_code != 0) { message(str_glue("Something went wrong with pulling submission {submission_id}")) } } else { message(str_glue("Submission {submission_id} is not a Docker submission")) } If you are pulling the submission metadata via a Submission View instead of using `syn.getSubmission()`, the annotation names will be in all lowercase (`dockerrepositoryname` and `dockerdigest`) instead of camel case, e.g. Python submissions_df = syn.tableQuery(...) for _, row in submissions_df.iterrows(): submission_id = row['id'] repo_name = row["dockerrepositoryname"] digest = row["dockerdigest"] ### Putting It All Together You now know how to query for new submissions and download them. By combining these steps, you can create a single script to evaluate all new submissions. Once a submission is evaluated, we recommend updating its status from `RECEIVED` so that it does not get picked up again if/when you re-query for new submissions in the future. Here's an example of an integrated script for downloading file submissions: **Python** Python """Evaluating file submissions.""" import os import pandas as pd import synapseclient syn = synapseclient.login() # Get new submissions. view_id = "syn53293336" submissions_df = ( syn.tableQuery(f"SELECT * FROM {view_id} WHERE status = 'RECEIVED'") .asDataFrame() .fillna("") ) # Evaluate each new submission. for submission_id in submissions_df["id"]: # Download predictions file to /path/to/download/. submission = syn.getSubmission(submission_id, downloadLocation="/path/to/download") with open(submission.filePath) as f: scores = ... # evaluate the predictions file # Update submission status to 'SCORED' if evaluation is successful, else 'INVALID'. submission_status_obj = syn.getSubmissionStatus(submission_id) submission_status_obj.status = "SCORED" if scores else "INVALID" syn.store(submission_status_obj) # File cleanup. try: os.remove(submission.filePath) except OSError as e: print(f"Could not delete predictions file for submission {submission_id}: {e}") **R** R library(synapser) library(tidyverse) synLogin() # Get new submissions. view_id <- "syn53293336" submissions_df <- synTableQuery(str_glue("SELECT * FROM {view_id} WHERE status = 'RECEIVED'")) %>% .$asDataFrame() %>% mutate(across(everything(), ~ ifelse(is.na(.) | . == "NaN", "", .))) # Evaluate each new submission. for (submission_id in submissions_df$id) { # Download predictions file to /path/to/download/. submission <- synGetSubmission(submission_id, downloadLocation = "/path/to/download") pred_file <- submission$filePath scores <- ... # evaluate the predictions file # Update submission status to 'SCORED' if evaluation is successful, else 'INVALID'. submission_status_obj = synGetSubmissionStatus(submission_id) submission_status_obj$status <- ifelse(!is.null(scores), "SCORED", "INVALID") synStore(submission_status_obj) # File cleanup. tryCatch({ file.remove(submission$filePath) }, error = function(e) { message(str_glue("Could not delete predictions file for submission {submission_id}: {e}")) }) } ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Submission status can only be set to specific values; refer to [SubmissionStatusEnum](https://rest-docs.synapse.org/rest/org/sagebionetworks/evaluation/model/SubmissionStatusEnum.html) for a list of acceptable values. *** ** * ** *** ## Assigning and Displaying Scores Scores are assigned to submissions by adding them as annotations. You can then display these scores in a Submission View by updating the view's schema to include these new annotations. ### Annotate Submissions You can only add annotations to submissions programmatically; the Synapse web UI does not currently support this feature. **Python** Python import synapseclient syn = synapseclient.login() # Get submission status object. submission_id = 123 submission_status_obj = syn.getSubmissionStatus(submission_id) # Add scores to the annotations metadata. ## adding one score submission_status_obj.submissionAnnotations["auc_roc"] = 0.0 ## adding multiple scores score_annots = { "auprc": 0.0, "pearson": 0.0 } submission_status_obj.submissionAnnotations.update(score_annots) # Save the new annotations. syn.store(submission_status_obj) **R** R library(synapser) synLogin() # Get submission status object. submission_id <- 123 submission_status_obj <- synGetSubmissionStatus(submission_id) # Add scores to the annotations metadata. ## adding one score submission_status_obj$submissionAnnotations$auc_roc <- 0.0 ## adding multiple scores score_annots <- list( auprc = 0.0, pearson = 0.0 ) submission_status_obj$submissionAnnotations$update(score_annots) # Save the new annotations. synStore(submission_status_obj) ### Display Scores To display the scores on Synapse: 1. Navigate to the Submission View containing the Evaluation Queue(s), and click on 3-bar icon (next to **Submission View Tools** ) to **Show Submission View Schema**. 2. The schema will now appear above the table. Click on **Edit Schema** and a new window will pop up. 3. Click **+ Add Column** . For "Column Name", enter the exact annotation key name you used (e.g. `auc_roc` from the code examples above). Update the "Column Type" appropriately. 4. Repeat Step 3 for each scoring metric you want to display. 5. Click **Save** to apply the changes. If done correctly, your Submission View should now include the new metric columns with scores displayed for each submission. **Troubleshooting:** If scores are not appearing, double-check that the column names in your schema exactly match the annotation keys on your submissions, including casing. For example, `AUC_ROC` is not considered the same as `auc_roc`. Another potential issue is a mismatch in the "Column Type". For instance, if you specify "Integer" but your values are strings, the scores will not display. *** ** * ** *** ## Tools for Automation If you're looking to automate the process of evaluating submissions as they come in, Sage offers several tools and services: ++Orchestrators++ * **ORCA (Paid Service):** This tool uses NextFlow to run workflows. Your main job is to provide the evaluation code (template available below) which the Data Processing \& Engineering (DPE) team then integrates into a workflow. For cost estimates, please contact the [DPE Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/5/group/7/create/51). * [**SynapseWorkflowOrchestrator**](https://github.com/Sage-Bionetworks/SynapseWorkflowOrchestrator): This tool executes Common Workflow Language (CWL) workflows, which you will design yourself (templates available below). This tool also requires manual setup and configuration to link it with your Challenge project. If you need help, contact the [Challenges \& Benchmarking Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/18/group/27/create/203). ++Workflow templates++ * [orca-evaluation-templates](https://github.com/Sage-Bionetworks-Challenges/orca-evaluation-templates) (for use with ORCA) * [data-to-model-challenge-workflow](https://github.com/Sage-Bionetworks-Challenges/data-to-model-challenge-workflow) (for use with the SynapseWorkflowOrchestrator) * [model-to-data-challenge-workflow](https://github.com/Sage-Bionetworks-Challenges/model-to-data-challenge-workflow) (for use with the SynapseWorkflowOrchestrator) --- language: "en" --- # Evaluation Queues An evaluation queue allows you to submit Synapse files or Docker images for evaluation. They are designed to support open-access data analysis and modeling challenges in Synapse. This framework provides tools for administrators to collect and analyze data models created by Synapse users for a specific goal or purpose. ## Create an Evaluation Queue To create a queue, you must first create a Synapse Challenge project and have edit permissions on an existing project. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For instructions on setting up a Synapse Project, see [Setting Up a Project](https://docs.synapse.org/synapse-docs/setting-up-a-project.md). If you do not see the challenge tab in your project: 1. Navigate to the bottom of the page and click **experimental mode: off**to turn on experimental mode. 2. Click **Project Tools** in the right corner and select **Run Challenge** and follow instructions. Once your project has the **Challenge** tab, navigate to it and click **Challenge Tools** in the right corner and select **Add Evaluation Queue**. ![create_evaluation_queues.png](https://docs.synapse.org/__attachments/a_286d8d33355f6a5b2d3cdf9dbe5fd8aca09b0e0de11a36f6f1884d86e62df7fc/create_evaluation_queues.png?cb=a3858a1bcfff8fab18070782f67555d1) An evaluation queue can take several parameters that you can use to customize your preferences. * Name: Unique name of the evaluation * Description: A short description of the evaluation * Submission instructions: Message to display to users detailing acceptable formatting for submissions * Submission receipt message: Message to display to users upon submission---the name of your evaluation queue MUST be unique, otherwise the queue will not be created ### Setting Quotas on an Evaluation Queue Optionally, you can restrict submissions by adding a quota to each round of your challenge. An evaluation queue can only have one quota. You must specify some required parameters: the length of time the queue is open, the start date, round duration, and number of rounds. It is optional to set a submission limit. * Duration (Round Start and Round end) - Select a date and time for the start and end of the round. * Submission Limit - The maximum number of submissions per team/participant per round. Please keep in mind that the system will prevent additional submissions by a user/team once they have hit this number of submissions. * Advanced Limits - You may set additional quotas for daily, weekly, or monthly submissions per team/participant. These limits can be combined by clicking the **+** sign next to the **Maximum Submissions** field. ### Share an Evaluation Queue Each evaluation has sharing settings, which limit who can interact with the evaluation. * **Administrator** sharing should be tightly restricted, as it includes authority to delete the entire evaluation queue and its contents. These users also have the ability to download all of the submissions. * **Can Score** allows for individuals to download all of the submissions * **Can Submit** allows for teams or individuals to submit to the evaluation, but doesn't have access to any of the submissions. * **Can View** allows for teams or individuals to view the submissions on a leaderboard. To set the sharing settings, go to the **Challenge** tab to view your list of evaluations. Click on the **Share** button per evaluation and share it with the teams or individuals you would like. **Important:** When someone submits to an evaluation queue, a copy of the submission is made, so a person with "Administrator" or "Can Score" access will be able to download the submission even if the submitter deletes the entity. ### Close an Evaluation Queue While there isn't technically a way of "closing" an evaluation queue, there are multiple ways to discontinue submissions for users. * Users are only able to submit to a queue if they have `can submit` permissions to it. If you have the ability to modify the permissions of a queue, you will still be able to submit to the queue due to your `administrator` access. * If the quota is set so the current date exceeds the the round start + round duration, no one will be able to submit to the queue. This includes users with administrator permissions. * Deleting a queue will also discontinue the ability to submit to it. Be careful when doing this, as deleting a queue is irreversible and you will lose all submissions. ## Submitting to an Evaluation Queue Any Synapse entity may be submitted to an evaluation queue. In the R and Python examples, you need to know the ID of the evaluation queue. This ID must be provided to you by administrators of the queue. The submission function takes **two optional parameters** : `name` and `team`. Name can be provided to customize the submission. The submission name is often used by participants to identify their submissions. If a name is not provided, the name of the entity being submitted will be used. As an example, if you submit a file named `testfile.txt`, and the name of the submission isn't specified, it will default to `testfile.txt`. Team names can be provided to recognize a group of contributors. ### Python Python import synapseclient syn = synapseclient.login() evaluation_id = "9610091" my_submission_entity = "syn1234567" submission = syn.submit( evaluation = evaluation_id, entity = my_submission_entity, name = "My Submission", # An arbitrary name for your submission team = "My Team Name") # Optional, can also pass a Team object or id #### R R library(synapser) synLogin() evaluation_id <- "9610091" my_submission_entity <- "syn1234567" submission <- synSubmit( evaluation = evaluation_id, entity = my_submission_entity, name = "My Submission", # An arbitrary name for your submission team = "My Team Name") # Optional, can also pass a Team object or id ## Submissions Every submission you make to an evaluation queue has a unique ID. This ID should not be confused with Synapse IDs which start with the prefix "syn" (for example, syn12345678). All submissions have a `Submission` and `SubmissionStatus` object. Navigate to a file in Synapse and click on **File Tools** in the upper right-hand corner. Select **Submit To Challenge** to pick the challenge for your submission. Follow the provided steps to complete your submission. ![submit_file_to_challenge.png](https://docs.synapse.org/__attachments/a_61c742abd17bdbd49e59f3ce001ebc46dd6c2c8b074e1351c927e1d588cc7224/submit_file_to_challenge.png?cb=d4a77a5a2dadf9663542281d418c653c) ### View Submissions of an Evaluation Queue Submissions can be viewed and shared with users through submission views creating dynamic leaderboards. Submission annotations can be added to a `SubmissionStatus` object and are automatically indexed in the view. ### Creating the Submission View Navigate to the **Tables** tab and under the **Table Tools** menu in the upper right-hand select **Add Submission View**: ![view_add_submission_view.png](https://docs.synapse.org/__attachments/a_cc6dce8498bf1df7c4f0da3b8a0f1090707e50c37119f28182f703925b7f3d52/view_add_submission_view.png?cb=9450d9122cc4667692f11fa498e12cf2) You can name the view, and select the evaluation queues to include in the scope. ![view_create_submission_view.png](https://docs.synapse.org/__attachments/a_3880fbfdf37a87fc35db43c74fe1c0d7223db862daa9bd15d0fbb1b1c27339f5/view_create_submission_view.png?cb=6edc567e0121e0fdc020df6ee9c0c1cc) You can add multiple evaluation queues to the scope: ![view_select_evaluation_queue.png](https://docs.synapse.org/__attachments/a_cc73be2b74be2863a2f4af9ffb5b3b80b18750b36c9d4747a0a87bb96bc6f5c5/view_select_evaluation_queue.png?cb=df0b25fef69582c2383922eda38dd162) **Note:** You must be an administrator of each selected evaluation queue to create the view. During the creation process the default columns for a submission view will be included: ![view_select_columns.png](https://docs.synapse.org/__attachments/a_1a208382d5c403f9ec02fb316db5d0097a73279cf33fb09e9f7bd0690127b668/view_select_columns.png?cb=04fc60f35e76faf6131a34a23acd33a3) Selecting **Add All Annotations** will automatically include all the annotations found on the submissions in the scope as columns for the view: ![view_add_annotations.png](https://docs.synapse.org/__attachments/a_f782932619898949510cc572028097c276602b6ee4489cdd51f5af4e42938d66/view_add_annotations.png?cb=ad54cee591a4683e14c5d6635440db32) ### Embed a Submission View in a Wiki Page Once created, a submission view can be embedded into a wiki page using the Synapse Table/View wiki widget: ![wiki_insert_table_query.png](https://docs.synapse.org/__attachments/a_ac07b5e58020ae22e938e8b19a0803f4e6c088f24d7636618d08ec59a4b64358/wiki_insert_table_query.png?cb=49781bf6371933a326a092b39c9844d5) You can input your own query statement such as `SELECT * FROM syn22155139 ORDER BY score DESC`. Remember, syn22155139 should be replaced with the synID of the submission view: ![wiki_insert_table_query_sql.png](https://docs.synapse.org/__attachments/a_651490682cc03bd79144b6521437b67ed4d19e32bee48ad835425ed0880b7b22/wiki_insert_table_query_sql.png?cb=d6389af8bf2096d6a1505531bfb1539b) ### Submit to an Evaluation Queue from a Wiki Page ![add_submission_widget.png](https://docs.synapse.org/__attachments/a_02be88413da3caae9135a4f16d56f5a4ff061de189f4dfab951985ad2bfde064/add_submission_widget.png?cb=77c5a64b3e058d43435a210b0199ad40) You may embed a**Submit To Evaluation**widget on a wiki page to improve visibility of your evaluation queue. The widget allows participants to submit to multiple evaluation queues within a project or a single evaluation queue. Currently, this wiki widget is required to submit Synapse projects to an evaluation queue. Synapse Docker repositories can not be submitted through this widget. ![submit_to_evaluation_widget.png](https://docs.synapse.org/__attachments/a_5ea81f1faac6cec20e1406b10370ed7bbaaa945d994b4b920235a18079868494/submit_to_evaluation_widget.png?cb=13f118b5f613f5305649db4f50cf2fbf) The "Evaluation Queue unavailable message" is customizable. A queue may appear unavailable to a user if: * The project doesn't have any evaluation queues. * The evaluation queue quota is set such that a user can not submit to the queue. * The user does not have permission to view a project's evaluation queues. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about sharing settings [here](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). --- language: "en" --- # FAQ ## About Synapse What does Synapse do? Synapse is a cloud-based data repository and sharing platform where researchers can share and describe content to co-analyze, learn from, and improve knowledge of health and disease. Synapse was developed to encourage research collaborations across institutional boundaries and is therefore provided as "Software As A Service" with a single instance used by all users. This makes it easy to discover and share public User Content, including Data, analyses, tools, methods, and other content. Synapse also supports private project spaces where the individual content contributor controls User Content sharing. Synapse provides a standard interface to describe User Content, where it comes from, and how to use it. Synapse also provides mechanisms for adding User Content and its descriptions. Synapse can facilitate sharing User Content stored in many locations, or cloud storage. This allows Synapse to store metadata about the Content, such as annotations, descriptive wiki pages, and provenance, but not the actual data. Currently, Synapse supports files stored in AWS S3 buckets and the Google Cloud Storage Platform. (see [Custom Storage Locations](https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html)). Is Synapse a data analysis platform? Not directly. Synapse helps you manage User Content, including Data, analysis, tools, methods, and results. However, using the programmatic interfaces built into Synapse makes it easy to set up analytical pipelines and *ad hoc* analyses that interact with Synapse. By default, Synapse uses Amazon's cloud infrastructure (S3) for storage, making it simple to allocate large compute resources and collocate them next to User Content storage. Synapse does support users performing analysis on data stored in S3 buckets in Synapse using the AWS Security Token Service (STS). (see [++Computing Directly on Data in Synapse In S3++](https://help.synapse.org/docs/Compute-Directly-on-Data-in-Synapse-or-S3.2048426057.html)) Who uses Synapse? Anyone 18 years or older may create an account on Synapse. Sage offers different plans for Synapse. These plans have varying restrictions and limitations on user training requirements and allowable user content. We have highlighted a series of [research communities](https://www.synapse.org/#!StandaloneWiki:ResearchCommunities) currently using Synapse for collaborative work and some [open resources](https://www.synapse.org/#!StandaloneWiki:OpenResearchProjects) hosted in Synapse. Can I get help using Synapse in my collaboration? Our [Discussion Forum](https://www.synapse.org/#!SynapseForum:default) is a great place to reach out to the broader Synapse community to find others that may be interested in a collaboration. Depending on your plan level ([Synapse Offerings](https://help.synapse.org/docs/Sage-Offerings.2965078125.html)), you may have access to our help desk. What are the Synapse Terms of Use? The [Terms and Conditions of Use](https://www.synapse.org/TrustCenter:TermsOfService)fully describes the governance terms and conditions of Synapse. In order to register on Synapse, you must review and agree to the terms of the Synapse Awareness and Ethics Pledge. For more information, see [Synapse Governance](https://help.synapse.org/docs/Synapse-Governance.2004255211.html). Is Synapse open source? Yes, Synapse is released under the [Apache 2.0 License](https://github.com/Sage-Bionetworks/Synapse-Repository-Services/blob/develop/LICENSE) The source code is available on [GitHub](https://github.com/Sage-Bionetworks/). Synapse is also offered free of charge as a hosted Software as a Service (SaaS) at . I am a developer. Is there API documentation? Yes, Synapse is built on top of a RESTful service that is automatically [documented](http://rest-docs.synapse.org/rest), including an [OpenAPI Specification](https://rest-docs.synapse.org/openapi/openapispecification.json). In addition, we have purpose-built APIs for [Python](https://python-docs.synapse.org/), [R](https://r-docs.synapse.org/), Java and a [command line](https://python-docs.synapse.org/build/html/CommandLineClient.html) interface. How do I set up my own instance of Synapse? Synapse was developed with the philosophy to encourage collaboration across institutional boundaries and is therefore provided as "Software As A Service" with a single instance used by all users. This makes it easy both to discover new content and share with new collaborators. We do support private project spaces where content sharing is controlled by the individual user. In addition, Synapse has the ability to reference resources that are stored elsewhere. This allows Synapse to store metadata about the content such as annotations, descriptive wiki pages and provenance but not the actual data. Currently Synapse has specific support for files stored at URLs, on SFTP servers, on AWS S3 and arbitrary file servers (see: [Custom Storage Locations](https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html)). What do I do if I find a bug? You may browse open issues or file a bug through our [Jira](https://sagebionetworks.jira.com/) tracker system. To file a bug, use the blue "Create" button in the top center of the page. Please be sure to include your email address in your submission so we may follow up with you. How do I get started? See [++Getting Started++](https://help.synapse.org/docs/Getting-Started.2055471150.html) for a breakdown of what you need to get started and how to make the most of Synapse. You may browse the public content catalog and access limited features, but to access most features of Synapse, you must [++register++](https://www.synapse.org/#!RegisterAccount:0) for a Synapse user account. Before uploading User Content, you will need to complete specific training demonstrating your understanding of the ethical, legal, and technical issues associated with using and sharing User Content and how User Content is managed and shared in Synapse. Why should I register for a Synapse account? You can browse public content in Synapse without registering. However, without an account, you cannot add new User Content to Synapse, download restricted files or tables, or access the most advanced features of Synapse. With an account, you can, among other things, create projects and wikis, download some open content, and request access to controlled User Content. Further, an account lets you collaborate with other Synapse users and create user teams. For more information, see the [++Account Types++](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html) page. What is a validated profile? Validating your profile is a process where your identity is established through a combination of your profile information, [ORCID](http://orcid.org/), and an external credential. Validation increases transparency between researchers and User Content donors. A validated profile is needed for access to specific User Content. Profile validation instructions can be found in the Settings tab of your Synapse profile page. Click the "Request Profile Validation" link for the required steps. ## Accessing Content My colleague put some content in Synapse. How do I find it? This depends on whether the content is public or private. If private, you will need to make sure your colleague has shared this content with you. Shared content is visible from your "Dashboard page" under the tab "Shared directly with me". If you favorite the content (using the star) it will appear under your list of favorites visible from the [Synapse toolbar](https://docs.synapse.org/synapse-docs/navigating-synapse.md) or on your [profile](https://docs.synapse.org/synapse-docs/managing-your-account.md). All public data is queryable. For more information see [help on querying](https://docs.synapse.org/synapse-docs/querying-tables-views-and-datasets.md) or from the "Search" box in the top right corner of any Synapse page. How do I find public datasets in Synapse? Multiple research communities use Synapse to generate data that is released to the public. A description of some of these communities can be found on the [Synapse Research Communities Page](https://www.synapse.org/#!StandaloneWiki:ResearchCommunities) and [public resources page](https://www.synapse.org/#!StandaloneWiki:OpenResearchProjects). Why should I register for a Synapse account? You can browse public content in Synapse without registering. However, without an account you cannot add new content to Synapse, nor can you upload or download files or tables. With an account, you can create projects and wikis, download open data and request access to controlled data. Further, an account lets you collaborate with other Synapse users and create user teams. For more information see the [Account Types](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md) page. What is a validated profile? Validating your profile is a process where your identity is established through a combination of your profile information, your [ORCID](http://orcid.org/), a signed Synapse Pledge, and an external credential. Validation increases transparency between researchers and data donors. A validated profile is needed for access to specific datasets and is currently required for access to data collected through Sage Bionetworks' [research apps](https://sagebionetworks.org/digital-health-assessments/). Profile validation instructions can be found in the Settings tab of your Synapse profile page. Click on the 'Request Profile Validation' link to see the required steps. ## Adding Content I'm ready to share data, how do I start? Synapse makes it easy to share files of any sort, with whomever you choose whether a small group of collaborators or the general public. You may share raw data, summarized data, analysis results, or anything in between. See [Uploading and Organizing Data](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data.md) for information and instructions on sharing your data. Why do I have to be a certified user to upload content? You must be a **certified use** r to post User Content on Synapse. To become a **certified user** , you must demonstrate an understanding of your responsibilities for sharing User Content through Synapse, especially *data derived from human participants* , by completing the required [training](https://www.synapse.org/#!Quiz:Certification). These responsibilities include ensuring that Data derived from human participants is de-identified (unless unambiguously authorized in writing) and that all applicable privacy laws and regulations are observed. To become a certified user, you will need to pass a brief [quiz](https://www.synapse.org/#!Quiz:Certification). Is everything I share on Synapse public? No. Use sharing settings to control who can see the content you create. By default, projects and their content are visible only to the user who created it. By using the Synapse sharing settings, you have the ability to grant other Synapse users, Synapse teams, or the public access to your Project content. You can learn more here: [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). Can I store sensitive information about human subjects in Synapse? It depends. You may not store sensitive information about human subjects in Synapse if you have a **Basic Plan** . If you have a **Self-Managed Plan** or a **Data Coordination Plan** , you may store sensitive information about human subjects in Synapse if authorized. Synapse has an IRB-approved data governance procedure that employs **Conditions for Use** to allow for the sharing of sensitive data in a controlled manner. You can learn more by reading [Sharing Settings, Permissions, and Conditions for Use](https://help.synapse.org/docs/Sharing-Settings,-Permissions,-and-Conditions-for-Use.2024276030.html). If you have questions or would like assistance in applying Conditions of Use to your User Content, please get in touch with the Synapse **Access and Compliance Team** at [act@sagebase.org](mailto:act@sagebase.org). Can I store User Content from outside the US in Synapse? It depends. If you have a **Basic Plan** , you may only upload user content that is anonymized or user content that is not subject to the data protection laws outside the United States, e.g., GDPR. If you have a **Self-Managed Plan** or a **Data Coordination Plan**and have reached an agreement with Sage regarding the storage of overseas data, you may store such data in Synapse. How do I know the content I put in Synapse will be secure? What security measures does Synapse have? Synapse stores content in Amazon Web Services, which provides a layer of security measures designed and implemented by [Amazon](https://aws.amazon.com/security/). While Synapse is an open access site, each user has control over who may access their content by using sharing settings. Where are my files stored? By default, Synapse stores files in Amazon Simple Storage Services (S3). However, it is possible to set up Synapse to store files in different locations in S3 or Google Cloud Storage. For files stored outside of S3, Synapse can be used to organize, manage, and access files through the use of Synapse annotations to store file-specific metadata. (see: [Custom Storage Locations](https://docs.synapse.org/synapse-docs/custom-storage-locations.md)) ## Sage Offerings What are Sage's Platform Offerings? Sage offers the following service plans: (1) a**Basic Hosting Plan** , (2) a paid-for **Self-Managed Plan** , and (3) a paid-for, customized,**Data Coordination Plan**. The Sage Terms of Service apply to each of these plans unless otherwise noted. Additional governance terms may apply. ### ++**Basic Hosting Plan**++ 1. Included Services The Basic Hosting Planis intended for users wanting to share small datasets for scientific, educational, research collaborations, and publications (including creating DOIs) purposes. This plan includes: * User Content hosting of up to 100GB of space * Self-service Project set-up with no direct support from Sage staff * A basic portal landing page (Wiki) * Users can create Projects with administrative control over who is granted access to each Project 2. User Content Longevity Sage will continue to host User Content in a Basic Hosting Plan for as long as the User Content is being viewed or accessed. Sage currently utilizes Amazon Web Services (AWS) Intelligent-Tiering storage in all service plans. Infrequently accessed User Content will be moved to access tiers that require longer retrieval times. Sage may, at its own discretion and with notice to the Project Administrator, implement cost mitigation strategies, including, for example, moving User Content to lower-cost hosting services or employing fee sharing if such User Content accumulates high egress charges. Sage reserves the right to delete User Content, including Private Content, if deemed inappropriate or if it conflicts with the Sage Terms of Service or at Sage's discretion, after 24 months of inactivity (i.e., User Content is not viewed or downloaded). Sage will reach out to the Project Administrator of the User Content at least two times (at the email address provided upon registration) warning that the User Content may be removed if the user does not respond with a proposed use case for the User Content. If the Project Administrator does not respond within 30 days of the second warning, Sage has the right to delete the User Content. 3. Third-party access to User Content Hosted in a Basic Hosting Plan Project Administrators are expected to respond to inquiries from Sage or other Synapse users without delay. If Sage receives complaints that a Project Administrator is not responding to inquiries, Sage will assume the Project Administrator's account is inactive and reserves the right to suspend the Project Administrator's account. ### ++**Self-Managed Plan**++ 1. Included Services The Self-Managed Plan includes: * Option 1: User Content hosting of up to 100GB of space. * Option 2: User Content hosting of up to 500GB of space. * Up to 15 hours of consulting services, including for setting up Projects and sharing User Content according to the F.A.I.R. principles and governance * Up to 25 hours of help desk support * Tools for self-managing User Content access requests * A basic portal landing page (Wiki) * Users can create Projects where they have administrative control over who is granted access to each Project and can deploy User Content restrictions by clickwrap agreements. 2. User Content Longevity Sage will keep User Content in a Self-Managed Plan for 5 years (or as otherwise negotiated). Sage utilizes Amazon Web Services (AWS) Intelligent-Tiering storage in all of the service plans. Infrequently accessed User Content will be moved to access tiers that require longer retrieval times. After the Self-Managed Plan expires, the Self-Managed Plan account holder will have 3 months to retrieve or make other arrangements with regard to their User Content or to renew the Self-Managed Plan. After this 3-month grace period, if the account holder does not retrieve or make other arrangements with regard to their User Content or renew the Self-Managed Plan, Sage reserves the right, at its sole discretion, to archive, freeze, delete, or move the User Content. 3. Third-Party Access to User Content Hosted in a Self-Managed Plan Project Administrators are expected to respond to inquiries from Sage or other Synapse Users without delay. If Sage receives complaints that a Project Administrator is not responding to inquiries, Sage will try to reach out to such Project Administrator, and to the institution associated with the Self-Managed Plan account. If the Project Administrator and institution are not responsive, Sage reserves the right to suspend the Self-Managed Plan account. Sage will not provide a refund if a Self-Managed Plan is suspended for failure to meet service level expectations. ### ++**Custom Plan**++ 1. Included Services The Data Coordination Plan offers customized, end-to-end management of User Content supporting collaborative, multi-institutional research consortia. This plan includes: * Customized consulting services and data curation, harmonization, and validation services * Expert resources to localize data policies and governance controls to your particular country's requirements as applicable * Services to create a customized and feature-rich data exploration portal for your coordination center that includes links to sophisticated computational environments 2. User Content Longevity The longevity of Sage's hosting of User Content will be mutually agreed upon by the account holder of the Data Coordination Plan and Sage. Because Data Coordination Plans are typically customized to meet the needs of a particular proposal, they are subject to additional contractual documentation. What are the payment terms? **Basic Hosting Plan**: Access to the Service is provided free of charge except in cases where users request additional services not included in the Basic Hosting Plan. Sage reserves the right to initiate fees for the Service, or portions thereof, at any time, by providing the user 30 days' prior written notice via Synapse or via the email provided during registration. If users do not wish to pay such fees, they can remove their User Content and terminate their account. Continued use of the Service after 30 days may trigger fee payment obligations. **Self-Managed and Data Coordination Plans**: These plans require users to pay fees. All fees are in U.S. Dollars and are non-refundable unless specified in the Sage Terms of Service or separate agreements. **Additional Services**: If users need additional services that extend beyond those specified under their respective plans, Sage may provide such services at an additional cost. Compliance with NIH Data Management and Sharing Policy (DMSP) The DMSP that went into effect on January 25, 2023 applies to all research funded or conducted in whole or in part by NIH. The goal of the policy is to promote the sharing of scientific data, which can accelerate biomedical research discovery, enable validation of research results, and provide accessibility to high-value datasets. Under the policy, NIH expects that investigators and institutions will: * Plan and budget for the managing and sharing of data * Submit a Data Management and Sharing (DMS) plan * Comply with the approved DMS plan and annually report on its implementation The **DMSP**provides guidance on how to manage and share data in a responsible and ethical manner to help accelerate biomedical research discovery and improve human health. ++**How can Sage help?**++ In most cases researchers must address the **NIH DMSP**when submitting their grant/funding proposal. For the Self-Managed and Data Coordination Plans, Sage can help: * Develop a data management and sharing plan that meets the requirements of the NIH Data Management and Sharing Policy. * Develop an appropriate budget for the data management and sharing plan that meets the F.A.I.R principles. * Provide guidance on how to store and secure the data at a level appropriate for its sensitivity. * Promote the responsible sharing of data for scientific, education, and research purposes. ++**How to get started?**++ Sage's expertise makes it easy to create an **NIH Data Management and Sharing** plan and budget. Sage can provide budgeting quotes, and text researchers can directly use in their NIH Data Management and Sharing plan. To start, [Contact Us](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/26/create/162). Can Sage help me plan and budget with respect to the NIH DMSP? If you have a Self-Managed or Custom Plan, Sage can help you prepare materials for your data management and sharing plan and budget. We provide quotes for services for use in your budget and standardized text for your plan. To start, [Contact Us](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/26/create/162). Where is my user content stored? For Basic plan users, by default, Synapse stores User Content in a shared bucket managed by Sage in Amazon Simple Storage Services (S3) in AWS US-EAST-1. Users can customize the locationby providing [++Custom Storage++](https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html). For Self-Managed and Data Coordination Plans, User Content can be stored in individual buckets managed by Sage in Amazon Simple Storage Services (S3) in AWS US-EAST-1. Users can customize the locationby providing [++Custom Storage++](https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html). I already have an account with data uploaded and shared, do I need to start paying? No, accounts and projects as of November 15, 2024 will continue under the terms of their previous agreement with Sage. What support does Synapse have for DOIs? You can mint DOIs for projects and files uploaded to Synapse, extremely useful for citing data. [Learn More](https://help.synapse.org/docs/Digital-Object-Identifiers-(DOIs).1972405096.html) My data is larger than the limits in the Basic and Self-Managed Plans. Is there a way to get more? These are the general storage limitations for each plan: up to 100GB for **Basic Hosting Plan** and up to 500GB for the **Self-Managed Plans** . You may pay for additional storage if needed. Please see our [Sage Offerings page](https://help.synapse.org/docs/Sage-Offerings.2965078125.html) for more information. There are no data limits if you use [Custom Storage](https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html)in your own cloud bucket, and depending on the dataset uploaded, storage may be sponsored by an existing project. [*Contact us*](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/26/create/162)*to learn more*. Can I add controls to my data in the Basic Plan? You may create a User Project where you have administrative control over who is granted access to the data within the project. However, you cannot implement specific User Content restrictions for your User Content under a **Basic Plan** . You must have a **Self-Managed Plan** or **Data Coordination Plan**to implement such User Content restrictions. Can I integrate my 3rd party workflow tools and compute environments with Synapse? Sage offers integrations with multiple cloud compute environments for customizedData Coordination Plans. Synapse is open source with standard interfaces and programming models (see [API Clients and Documentation](https://help.synapse.org/docs/API-Clients-and-Documentation.1985446128.html)) How do I control access to my content? You can manually add or remove Synapse users to your project spaces in the **Basic Plan** . In the **Self-Managed Plan** , Sage can set up User Content restrictions and require accessors to record their agreement to comply with such restrictions through a clickwrap agreement prior to accessing the User Content. For the **Data Coordination Center Plan**, we offer a complete data access committee service, either managed by you or by Sage to review/administer requests to controlled-access data. What happens to the content at the expiration of the service plan's period? After the **Self-Managed Plan** expires, the **Self-Managed Plan** account holder will have 3 months to retrieve the User Content or to renew the plan. After this 3-month grace period, Sage reserves the right to delete the User Content. For the **Data Coordination Plan** , the longevity of Sage's hosting of User Content and any transition period will be mutually agreed upon by the account holder of the **Data Coordination Plan**and Sage. Can I have a custom Portal for my project? Sage offers bespoke portals for customized **Data Coordination** **Plan** . For the **Basic** and **Managed Plans**, you can create your own lightweight portal directly in Synapse. What is the definition of "de-identified" User Content? "Deidentified Data" refers to Personal Data from which all directly identifiable elements (e.g., name, street address, date of birth, government identity number, etc.) is removed and the individual is solely identified by a random, unique reference number or code that is not derived from or related to the individual's personal information. --- language: "en" --- # Finding and Downloading Data You can find and download files from the Synapse web interface or by using one of the [programmatic clients](https://docs.synapse.org/synapse-docs/api-clients-and-documentation.md). If you are downloading files from the web, you can either download single files at once, or you can add multiple files to a download cart, which is similar to an online shopping cart. There is a catch with downloading from the web---the maximum size of a download is 5 GB, or a maximum of 100 files if using the download cart. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For instructions on downloading from the web, see [Downloading Data From the Synapse UI](https://docs.synapse.org/synapse-docs/downloading-data-from-the-synapse-ui.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For instructions on downloading larger files in greater numbers using one of our programmatic clients, see [Downloading Data Programmatically](https://docs.synapse.org/synapse-docs/downloading-data-programmatically.md). ## Finding Data Before you can download data of interest, you need to find it. Do so by navigating to the project, file, or folder that you wish to download, either through the [projects list](https://help.synapse.org/docs/Navigating-Synapse.2048557182.html#NavigatingSynapse-Projects), or by using the [global search bar](https://help.synapse.org/docs/Navigating-Synapse.2048557182.html#NavigatingSynapse-Search). Once you're in a project, click the **Files**tab. This is where you'll find all the files for this project, possibly nested within folders. ## Finding a Subset of Data Another, more specific, way of finding data of interest is by searching tables, views, or datasets. This can allow you to find data that is spread across multiple folders or projects, to find just the right subset of data that you care about. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information on this at [Querying Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/querying-tables-views-and-datasets.md). ## Accessing Data Before you can download a file from Synapse, you must determine whether you have access to it. Click on a file name and look for the **Access**status, followed by: * an unlock (🔓) icon - you have access, OR * a lock (🔒) icon - you don't have access To gain access, contact the project owner or click **Request Access**. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more about data access, see [Data Access Types](https://docs.synapse.org/synapse-docs/data-access-types.md). ## Downloading Data Next, learn how to download data from the [Synapse UI](https://docs.synapse.org/synapse-docs/downloading-data-from-the-synapse-ui.md) or [programmatically](https://docs.synapse.org/synapse-docs/downloading-data-programmatically.md). --- language: "en" --- # Getting Started Welcome to the Synapse docs site! This site exists to help you make the most of your experience using Synapse. Whether you're new to Synapse, or an experienced user, this docs site should help. ## What's on this docs site for you? * Read [about Synapse](https://docs.synapse.org/synapse-docs/about-synapse.md)---how it got started and how it fits into the bigger data-sharing picture * Gain a better understanding of [Sage Bionetworks](https://sagebionetworks.org/) (that's us---the nonprofit organization that created Synapse) and our other platforms that coincide with Synapse (such as portals) * Learn about [Synapse governance](https://docs.synapse.org/synapse-docs/synapse-governance.md) and how it protects data privacy * Get familiar with the structure and components of Synapse so you can [navigate it with ease](https://docs.synapse.org/synapse-docs/navigating-synapse.md) and [manage your account](https://docs.synapse.org/synapse-docs/managing-your-account.md) * Learn how to [upload and organize your data](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data.md) * Learn how to [curate your data](https://docs.synapse.org/synapse-docs/curating-data.md), as well as other data stored in Synapse * Learn how to [find and download data](https://docs.synapse.org/synapse-docs/finding-and-downloading-data.md) of interest * Get information on our [API clients](https://docs.synapse.org/synapse-docs/api-clients-and-documentation.md), including installation instructions and access to API documentation * Put instructions into action with our [use cases](https://docs.synapse.org/synapse-docs/use-cases.md)---real applications of specific tasks * Look up an unfamiliar term or acronym in our [glossary](https://docs.synapse.org/synapse-docs/glossary.md) * See our [help section](https://docs.synapse.org/synapse-docs/help.md) for further assistance via the FAQ page, discussion forum, or contact information to get in touch * Plus, much more---browse our articles or use the search bar to find exactly the information you need ## What do you need to get started? First off, to browse Synapse and access its most basic level of functionality, you will need to [register for a Synapse account](https://www.synapse.org/#!RegisterAccount:0). In doing so, you will be agreeing to the [Synapse Code of Conduct and Synapse Terms and Conditions of Use](https://docs.synapse.org/synapse-docs/synapse-governance.md). If you're simply looking to view a public wiki page, or browse our public project or file catalog, you can do so without registering for a Synapse account. To perform other functions within Synapse, such as exploring, downloading, uploading, and organizing data, there are additional steps to take: ### Exploring and downloading data As a [registered user](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html#SynapseUserAccountTypes-RegisteredUser), you can view and download any data that is publicly available. However, some data is considered controlled access and requires additional protections for who can access it and how it can be used. This data will be clearly labelled with its specific [conditions for use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). Occasionally, access to controlled data requires additional steps, like having your analysis plan approved by an ethics board or IRB. Controlled access data can not be shared. Anyone wishing to gain access to said data must individually agree to the conditions for use. ### Uploading, organizing, and curating data To work with data of your own or your team's, you must become a [certified user](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html#SynapseUserAccountTypes-CertifiedUser). To become certified, you must pass a [certification quiz](https://www.synapse.org/#!Quiz:Certification), which will ask you about information that can be found throughout this documentation site. ### Accessing mHealth data To access mHealth data, you must become a [validated user](https://help.synapse.org/docs/Synapse-User-Account-Types.2007072795.html#SynapseUserAccountTypes-ValidatedUsers). --- language: "en" --- # Glossary Browse the glossary to learn more about terms and definitions commonly used throughout Synapse. ## ACT Abbreviation for the Synapse Access and Compliance Team, a group of people who are responsible for setting, maintaining, and controlling governance throughout Sage Bionetworks and its platforms. The ACT recommends appropriate safeguards depending on the type of data and who can access it, and can apply certain limitations or conditions for data access based on how the data will be shared. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information on the ACT and Synapse governance [here](https://docs.synapse.org/synapse-docs/synapse-governance.md). *** ** * ** *** ### Annotations Annotations help users search for and find data, and they are a powerful tool used to systematically group and/or describe things in Synapse. Annotations are stored as key-value pairs in Synapse, where the key defines a particular aspect of your data (for example, species, assay, file format) and the value defines a variable that belongs to that category (mouse, RNAseq, .bam). You can use annotations to add additional information about a project, file, folder, table, or view. Annotations can be based on an existing ontology or controlled vocabulary, or can be created as needed and modified later as your metadata evolves. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn about annotating data [here](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md). *** ** * ** *** ### Anonymous access This is a type of data access setting---data set as anonymous access is available for anyone on the web, without conditions for use. The other data access tiers are: private access, controlled access, and open access. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about data access [here](https://docs.synapse.org/synapse-docs/data-access-types.md). *** ** * ** *** ### API Abbreviation for Application Programming Interface, this is a type of software application that allows for the integration or connection between otherwise unconnected services. Synapse uses API clients to allow for the programmatic use of certain tasks, such as uploading and downloading data. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information, including installation instructions, on Sage's API clients [here](https://docs.synapse.org/synapse-docs/api-clients-and-documentation.md). *** ** * ** *** ### Certified User This is one of four user account types in Synapse, which determines what actions a user can perform. Certified users have full access to Synapse functionality. The other user account types are: anonymous users, registered users, and validated users. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information on user account types [here](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md). *** ** * ** *** ### Challenges An open-science, collaborative competition framework for evaluating and comparing computational algorithms. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about Challenges [here](https://docs.synapse.org/synapse-docs/challenges.md). *** ** * ** *** ### Command line One of the API clients that provides a way to use Synapse programmatically. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to install the Synapse command line client [here](https://help.synapse.org/docs/Installing-Synapse-API-Clients.1985249668.html#InstallingSynapseAPIClients-CommandLine). *** ** * ** *** ### Controlled access This is a type of data access setting---data set at as controlled access is available to registered, certified, or validated users that fulfil specific requirements for data access. The other data access tiers are: private access, open access, and anonymous access. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about data access [here](https://docs.synapse.org/synapse-docs/data-access-types.md). *** ** * ** *** ### Data dictionary A repository or visual representation for metadata/schemas used within a given community/project. *** ** * ** *** ### Data science Data science is an interdisciplinary field that uses scientific methods, processes, algorithms, and systems to extract knowledge and insights from structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains. *** ** * ** *** ### Data model The representation of metadata terms and their attributes for a given research and/or project ecosystem. A data model explicitly determines the structure of data. *** ** * ** *** ### Digital object identifier (DOI) A distinct alphanumeric string assigned to uniquely label and identify a digital object. A DOI is defined by a digital location like a URL and a description of the object, which includes attribution and a creation or publication date. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about DIOs [here](https://docs.synapse.org/synapse-docs/digital-object-identifiers-dois.md). *** ** * ** *** ### Docker A tool for creating, running, and managing lightweight virtual machines to bundle code and other dependancies. You can add a Docker container to a project and share it with your teammates. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about Docker [here](https://docs.synapse.org/synapse-docs/synapse-docker-registry.md). *** ** * ** *** ### Entity Any distinct item in Synapse that has its own synID, including a file, folder, project, wiki, dataset, and view, among other items. *** ** * ** *** ### Experimental mode A mode in Synapse where new features and feature updates that are still in development until they are ready to be pushed live. Anyone can test out this mode using the **Experiment Mode**link at the bottom right of Synapse. *** ** * ** *** ### FAIR Abbreviation for Findable, Accessible, Interoperable, and Reusable. This is the standard set of principles that we follow at Sage. FAIR data are discoverable to users through precise metadata, understandable in terms of how the data can be used, machine-readable to enable computational analysis, and ultimately, fit for reuse. *** ** * ** *** ### Governance A set of conditions and responsibilities that Sage has established to enable the ability of our organization to ensure quality, compliance, and usability of data uploaded to our platforms. The resources and policies implemented in Sage's governance include novel approaches to informed consent, clinical protocols, and data access. Synapse governance is an essential component of the Synapse platform; it is a system of policies, procedures, and tools for managing and protecting data in Synapse. Our policies define the community norms, user rights, and user responsibilities. Our procedures determine how to and who can contribute, access, and use content. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about Synapse governance [here](https://docs.synapse.org/synapse-docs/synapse-governance.md). *** ** * ** *** ### JSON Abbreviation for JavaScript Object Notation. JSON is a data-interchange format or language based on two structures: an object and associated values, or an array. *** ** * ** *** ### JSON schema A specific JSON-based format that defines the structure of JSON data for validation, documentation, and interaction control. It provides a contract for the JSON data required by a given application, and how that data can be modified. *** ** * ** *** ### Key-value pairs Key-value pairs are used in annotations, where the key defines a particular aspect of your data (for example, species, assay, file format) and the value defines a variable that belongs to that category (mouse, RNAseq, .bam). *** ** * ** *** ### Manifest This is a file that gets uploaded alongside data, which specifies information about the data files being uploaded. It also contains annotations that will be associated with the file in Synapse. It tells the computer the current directory of the file to be uploaded (via `path`) and the Synapse ID of the folder where files will be uploaded (via `parent`). The manifest can also be used to describe provenance of each file, indicating how it was generated, but this is optional (but helpful). There are several different types of manifests used throughout Synapse: * *Upload manifest* : This is a .tsv file used to upload metadata---more details, along with a template, are provided [here](https://help.synapse.org/docs/Upload-and-Download-Data-in-Bulk.2003796248.html#UploadandDownloadDatainBulk-UploadingDatainBulk). * *Download manifest:*This is used when downloading data programmatically---the template is provided by Synapse Python Client. * *File Schema Driven Manifest:*This is based on the new File Schema. * *Portals Manifest:*This is currently provided when exporting data. *** ** * ** *** ### Markdown Markdown is a simple language used for creating formatted text (such as italics, bold, hyperlinks, paragraph breaks, etc.). It is simple in that it is readable to computers while also being appealing and understandable to people. Markdown is used in certain parts of Synapse, including [wikis](https://docs.synapse.org/synapse-docs/creating-and-managing-wikis.md) and [discussion forums](https://docs.synapse.org/synapse-docs/discussion-forums.md). *** ** * ** *** ### Metadata Metadata is additional, standardized information included alongside the data to give it context---*data about the data*, if you will. Metadata is what allows data in Synapse to be searchable, discoverable, accessible, re-usable, and understandable to others, including those who were not involved in the data generation process. Metadata can be descriptive (i.e., the name of the file), administrative (i.e., provenance information), or research-based (i.e., information about the sampling and handling of data). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about [Annotating Data With Metadata](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md). *** ** * ** *** ### Ontology In the context of data science, an ontology is essentially the system in place for naming and classifying entities and the relationships between them, as they exist in a particular data model. For example, the ontology of a research study would specify an appropriate naming convention for terms used throughout the study. *** ** * ** *** ### Open access This is a type of data access setting---data set at as open access is available to all registered Synapse users, without use limitations. The other data access tiers are: private access, controlled access, and anonymous access. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about data access [here](https://docs.synapse.org/synapse-docs/data-access-types.md). *** ** * ** *** ### Permissions This refers to the level of access that a Synapse user or team has to view, download, edit, delete, and manage data. Permissions can be set within the sharing settings of a project, files, folders, and tables. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about permissions and how to set them [here](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). *** ** * ** *** ### Private access This is a type of data access setting---data set at as private access is visible only to the data owner and any other users who the owner grants access to. The other data access tiers are: controlled access, open access, and anonymous access. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about data access [here](https://docs.synapse.org/synapse-docs/data-access-types.md). *** ** * ** *** ### Project In Synapse, projects act as containers that group relevant content and people together. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about projects [here](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data-into-projects-files-and-folders.md). *** ** * ** *** ### Provenance Provenance is a concept describing the origin of something. In Synapse, it is used to describe the connections between the workflow steps used to create a particular file or set of results. In other words, it tracks the relationship between data, code, and analytical results. The Synapse provenance system is one of many solutions that makes research work reproducible by you and others. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Read more about Synapse provenance [here](https://docs.synapse.org/synapse-docs/provenance.md). *** ** * ** *** ### Python One of the API clients that provides a way to use Synapse programmatically. Learn how to install the Synapse python client [here](https://help.synapse.org/docs/Installing-Synapse-API-Clients.1985249668.html#InstallingSynapseAPIClients-Python). *** ** * ** *** ### Registered User This is one of four user account types in Synapse, which determines what actions a user can perform. Registered users can create projects and wikis, collaborate with other registered users and create Synapse teams, can download publicly available data, and can access controlled data (if they fulfil the conditions for use) The other user account types are: anonymous users, certified users, and validated users. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information on user account types [here](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md). *** ** * ** *** ### RNA-Seq A sequencing technique which uses next-generation sequencing to reveal the presence and quantity of RNA in a biological sample at a given moment, analyzing the continuously changing cellular transcriptome (the set of all RNA transcripts in an individual or population of cells). *** ** * ** *** ### Schema A snapshot of all the objects contained in a database and their relationship. Essentially, it is the structure of your data. In Synapse entities, such as views and tables, the schema defines the column names, as well as the values or types of data allowed in each column. *** ** * ** *** ### Sharing Settings Determine who can access content in Synapse and what permissions those users have with respect to a dataset. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about sharing settings and permissions [here](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). *** ** * ** *** ### Synapse ID (synID) Every object in Synapse (file, folder, project, table, view, user, etc.) is designated a unique Synapse ID (also known as synID) that is readable by programmatic clients. *** ** * ** *** ### User Interface (UI) The front-end or public-facing version (as opposed to the back-end) of a website, platform, or app, such as Synapse and the various Sage portals. *** ** * ** *** ### Validated User This is one of four user account types in Synapse, which determines what actions a user can perform. Validated users are certified users that have applied to have their user profile validated. This validation makes you eligible to request access to mHealth data. The other user account types are: anonymous users, registered users, and certified users. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Find more information on user account types [here](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md). --- language: "en" --- # Help This documentation site is designed to help you use Synapse. If you've explored the site and find you still have unanswered questions, try these other options: ## FAQ Browse through our [frequently asked questions](https://docs.synapse.org/synapse-docs/faq.md)---your question may be a common one! ## Synapse help forum [The forum](https://www.synapse.org/#!SynapseForum:default) is central place for Synapse users to pose questions, from general inquiries to specific troubleshooting issues. Sage monitors and maintains this help forum and answers questions on a regular basis. External Synapse users are welcome and encouraged to answer other users' questions as well. If you have a question of your own, feel free to browse the forum first to see if your question has already been asked and answered. ## Discussions Similar to the help forum, every Synapse project has a **Discussion**tab, which provides a space for team members to collaborate, ask questions, and even invite external colleagues to contribute. ## Contact us Have you tried all of the options above and still need help? Contact us at our [on-line help desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9) and we'll get back to you as fast as we can. --- language: "en" --- # Installing Synapse API Clients API clients provide a way to use Synapse programmatically. This page shows you how to install a client and login to Synapse programmatically. Packages to interface and access data within Synapse are available for: * Command line * Python * R To manage stored login credentials, see [Client Configuration](https://docs.synapse.org/synapse-docs/client-configuration.md). ## Command Line The Synapse command line client is implemented in Python and comes with the Synapse Python package. To install the Synapse command line client, make sure that you have Python and pip installed. For more information, see the [Python](https://www.python.org/downloads/) and [pip](https://pip.pypa.io/en/stable/installing/) installation instructions. After installing Python and pip, open your terminal (Mac OS or GNU/Linux) or Command Prompt (Windows), and run the following command: Python pip install synapseclient synapse login -p "authtoken" synapse -h For more documentation on the command line client, see the [synapseclient docs](https://python-docs.synapse.org/tutorials/command_line_client/). ## Python The `synapseclient` package is available to use Python to access Synapse. See the [Python client docs](https://python-docs.synapse.org/) for supported versions of Python. Python # Run pip install in your terminal first # pip install synapseclient import synapseclient syn = synapseclient.login(authToken="authtoken") For complete documentation of the Python client, visit the [synapseclient docs](https://python-docs.synapse.org/). ## R The `synapser` package is available for R versions 4.1 and above. R install.packages("synapser", repos=c("http://ran.synapse.org", "https://cloud.r-project.org")) install.packages("synapser") library(synapser) synLogin(authToken="authtoken") For more documentation on the R client, see [synapser R docs.](https://r-docs.synapse.org/) ## To Report an Issue Report client-specific issues or bugs at the [R client](https://github.com/Sage-Bionetworks/synapser/issues) or [Python (including the command line client)](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/16/create/206) issues pages. If you have questions on how to use the clients to interact with Synapse, visit the [Synapse Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9). --- language: "en" --- # Integrating Synapse With Your RNA-Seq Workflow The goal of this tutorial is to demonstrate how Synapse can manage files and track processing steps in an RNA-Seq workflow. ## Getting Raw Data The first step is to download the data onto the computer where you will be processing it. The data used in this example is a small RNA-Seq dataset for adrenal and brain tissue generated by the [Illumina Body Map](https://www.ebi.ac.uk/arrayexpress/experiments/E-MTAB-513/) project. A sub-sampled raw dataset has been stored in a public [Synapse project](https://www.synapse.org/#!Synapse:syn2468548/files/). Here, we access two `fastq` files and a small region of chr19 (300000-350000 Mb) of the hg19 reference genome directly by their Synapse identifiers and download them to our local computer. # Get brain fastq file synapse get syn2468554 # Get adrenal fastq file synapse get syn2468552 ## Map the Raw Reads Now that we have the data, you can use the alignment tool of your choice to map these reads. We will use [STAR](http://bioinformatics.oxfordjournals.org/content/early/2012/10/25/bioinformatics.bts635) to map the reads. ## Setting Up the Local Environment mkdir demo-rnaseq-workflow #dir to store the STAR genome index for the reference genome mkdir ref-genome ## Creating a Genome Index # the reference genome # downloads as hg19_chr19_subregion.fasta synapse get --downloadLocation ref-genome/ syn2468557 #create a STAR genome index STAR --runMode genomeGenerate --genomeDir ref_genome --genomeFastaFiles ref-genome/hg19\_chr19\_subregion.fasta ## Map Adrenal and Brain Tissue Reads star --runThreadN 1 --genomeDir ref-genome/ --outFileNamePrefix brain --outSAMunmapped Within --readFilesIn brain.fastq star --runThreadN 1 --genomeDir ref-genome/ --outFileNamePrefix adrenal --outSAMunmapped Within --readFilesIn adrenal.fastq Store the results and provenance in Synapse. # create a project synapse create Project --name demo-rnaseq-workflow # Use Synapse ID reported from the above command to use as the parent ID synapse store brain.sam --parentId syn234567890 --used brain.fastq ref-genome/hg19\_chr19\_subregion.fasta synapse store adrenal.sam --parentId syn234567890 --used adrenal.fastq ref-genome/hg19\_chr19\_subregion.fasta *** ** * ** *** **Need More Help?** Ask a question in the Synapse [Help Forum](https://www.synapse.org/#!SynapseForum:default). Your feedback is key to improving our documentation, so [contact us](mailto:synapseinfo@sagebase.org) if something is unclear or open an [issue](https://sagebionetworks.jira.com/secure/CreateIssue.jspa?issuetype=3&pid=12124). --- language: "en" --- # JSON Schemas In Synapse, you can streamline the annotation process and ensure that your metadata meets certain requirements using JSON Schemas. You can define a JSON Schema to require certain fields, restrict annotations to specific values, and even apply conditional logic to validate metadata. JSON Schemas primarily supplement [annotations](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md); you should understand annotations in Synapse before using JSON Schemas. This document also assumes you are comfortable with using the [Synapse Python Client](https://python-docs.synapse.org/). ## JSON Schemas and Annotations JSON Schema is a tool used to validate data. In Synapse, JSON Schemas can be used to validate the metadata applied to a project, file, folder, table, or view, including the Annotations applied to it. To learn more about JSON Schemas, check out [JSON-Schema.org](http://json-schema.org/). Synapse supports a subset of features from json-schema-draft-07. To see the list of features currently supported, see the [JsonSchema object definition](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/schema/JsonSchema.html) from our REST API Documentation. When a JSON Schema is bound to an object in Synapse, a couple of things happen: * When the metadata or schema changes, the metadata is automatically validated against the applied JSON schema. * In the web UI, a custom form is shown when editing Annotations to help write Annotations that match the bound schema. ## Organizations JSON Schemas are managed by Organizations. At this time, Organizations must be created via a programmatic client or REST API call. Organizations are different from [teams](https://docs.synapse.org/synapse-docs/teams.md), which can be used for collaboration, communication, and data sharing. ## Create an Organization To create an Organization, all you need is a name, which must meet [certain requirements](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/schema/CreateOrganizationRequest.html). In Python, after logging in, you can create an organization. Note that you'll have to change the organization name to something unique. Python import synapseclient from synapseclient.models import SchemaOrganization syn = synapseclient.login() organization_name = "SynapseDocs" organization = SchemaOrganization(name=organization_name) organization.store() ## Create a JSON Schema Once you've created an Organization, you can create a JSON Schema. We'll create a simple schema that specifies an annotation called "color". Note that you will have to modify the organization name in the schema $id to successfully create your own schema. All JSON Schemas published to Synapse are publicly viewable by anyone on the internet, so make sure your schemas don't include sensitive information. Python from synapseclient.models import JSONSchema schema_body = { "$schema": "http://json-schema.org/draft-07/schema#", "$id": "https://repo-prod.prod.sagebase.org/repo/v1/schema/type/registered/SynapseDocs-Color", "properties": { "color": { "type": "string", "title": "Color", "description": "The color of the object", "enum": [ "Red", "Green", "Blue", "Yellow", "Orange", "Purple", "Brown", "Black", "White" ] } }, "required": [ "color" ] } schema = JSONSchema(name="SynapseDocs-Color", organization_name=organization_name) schema.store(schema_body=schema_body, version=VERSION) ### Schema Versioning You can create new versions of the schema by issuing a new request to the same endpoint, POST /schema/type/create/async/start. When you bind a JSON schema to an object, you can choose to bind a particular version of the schema to prevent updates to the schema from applying to the object. ## Bind a JSON Schema to an Object You can bind a JSON Schema to any project, folder, file, table, or view. When you bind a JSON Schema to a project or folder, then all items inside of the project or folder will inherit the schema binding, unless the item has a schema bound to itself. Only one schema can be bound to an item at a time. Bound schema inheritance is similar to [Sharing Settings](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md) inheritance, but is tracked separately. If you have edit access on a Synapse object, you can bind a schema to the entity in Python: Python from synapseclient.models import Folder objectId = 'syn########' # Replace the ID with your own, it can be any Synapse Entity folder = Folder(id=objectId).get() folder.bind_schema( json_schema_uri=schema.uri, enable_derived_annotations=True ) Even though only one schema can be applied to an item, you can use JSON schema references to create a schema composed of multiple sub-schemas. In the `jsonSchemaObjectBinding`, you may also include the boolean property `enableDerivedAnnotations` to have Synapse automatically calculate derived annotations based on the schema. See the Derived Annotations section below for more information. ## Annotate an Object with a Schema Navigate to the file or folder for which you've bound a schema. As you edit the annotations on the file, you will see a form that corresponds to the schema that you have bound. ![image-20211007-204426.png](https://docs.synapse.org/__attachments/a_bcfa66f718f9cd2a83ea4d8c28ffb991bda8236a51390f7b2af816acb3f896ed/image-20211007-204426.png?cb=389754c292e0c9628e36db73b6bfbb3a) You'll also be able to see if a file's metadata or annotations are invalid because of missing or invalid data. ![image-20211007-204941.png](https://docs.synapse.org/__attachments/a_df1983d9a13603ec7ad2ba860a66a0cc1673b3d2f37459706b832edbb66f8a91/image-20211007-204941.png?cb=48dd6700234f20f576769917d0c2393a) Validation Status: Missing ![image-20211007-205101.png](https://docs.synapse.org/__attachments/a_6412306929cea3eef51d7c6e13f016043159d1953f23e879cb1221dbc9ff15ea/image-20211007-205101.png?cb=36420cafa9c8449b1473531500146851) Validation Status: Invalid ![image-20211007-205131.png](https://docs.synapse.org/__attachments/a_9b8db08468edb6e64952656fe35b3ed0d3396fe55d0dbbc98ee72de1362e7c90/image-20211007-205131.png?cb=9cab7217e52910b464ed19e99216fe23) Validation Status: Valid ## Derived Annotations JSON Schemas can also be used to prescribe default annotation values. The annotations can be static, or based on conditional properties. ### Defining Derived Annotations using a JSON Schema Derived annotations can be enabled for a set of objects in Synapse when the schema is bound to the object. The `JsonSchemaObjectBinding` request object should contain the `enabledDerivedAnnotations` property with a value of `true`. The contents of the bound JSON Schema will be used to determine the derived annotations. Derived annotations are denoted using the JSON Schema keywords `default` and `const`. For example, consider the following JSON Schema: JSON { "type": "object", "properties": { "derivedFromConst": { "type": "string", "const": "Derived Constant Value" }, "derivedFromDefault": { "type": "string", "default": "Derived Default Value" } } } Any object that has this schema will have the following derived annotations: | **Annotation Key** | **Value** | |--------------------|------------------------| | derivedFromConst | Derived Constant Value | | derivedFromDefault | Derived Default Value | ### Conditionally Derived Annotations JSON Schemas can also be written to conditionally apply annotations. For example, consider the following JSON Schema: JSON { "type": "object", "properties": { "country": { "type": "string", "enum": [ "United States", "Canada" ] }, "measurementSystem": { "type": "string" } }, "if": { "properties": { "country": { "const": "United States" } }, "required": [ "country" ] }, "then": { "properties": { "measurementSystem": { "const": "Imperial" } } }, "else": { "properties": { "measurementSystem": { "const": "Metric" } } } } On an object where this schema is bound, the derived annotation value for `measurementSystem` of an object would depend on the actual annotation value of `country`. ### Viewing Derived Annotations Derived annotations will be returned when fetching annotations using the [GET /entity/{id}/annotations2 API](https://rest-docs.synapse.org/rest/GET/entity/id/annotations2.html) by setting the `includeDerived` parameter to true. Derived annotations will be included with other entity metadata when using the [GET /entity/{id}/json](https://rest-docs.synapse.org/rest/GET/entity/id/json.html) service by setting the `includeDerivedAnnotations` parameter to true. A table or view can be updated to show derived annotations using the [POST /column/column/view/scope/async/start API](https://rest-docs.synapse.org/rest/POST/column/view/scope/async/start.html) by passing `includeDerivedAnnotations` in the request body with a value of `true`. ### Limitations of Derived Annotations * Conditionally derived annotations cannot be derived from other conditional annotations * Derived annotations will only be generated when the annotation data is valid against the bound JSON Schema --- language: "en" --- # Linking to External Repositories and Registries with Bioregistry Synapse supports linking to external biological databases, repositories, and registries using standardized [http://bioregistry.io](http://bioregistry.io/) prefixes. This feature allows you to create clickable links in Tables, Views, and Portals that automatically resolve to external resources, making it easier to reference data across the broader research ecosystem. ## What are Bioregistry Prefixes? The Bioregistry is an open source, community-curated registry that provides standardized prefixes for biological databases and repositories. These prefixes allow you to create compact identifiers (CURIEs) in the format `prefix:identifier` that can be automatically resolved to the corresponding external resource. For example: * `pubmed:38117484` is "[The Epigenetic Evolution of Glioma Is Determined by the IDH1 Mutation Status and Treatment Regimen,](https://bioregistry.io/pubmed:38117484/)" by Malta, Sabedot, *et. al*. * `uniprot:Q8N653` is the [Uniprot entry for the human transcriptional regulator LZTR1](https://bioregistry.io/uniprot:Q8N653) * `geo:GSE141509` links to a [Gene Expression Omnibus microarray dataset](https://bioregistry.io/geo:GSE141509) examining a mouse model of late-onset Alzheimer's disease ## How It Works in Synapse When you include a bioregistry prefix and identifier in a Table or View cell, Synapse automatically converts it into a clickable link that opens the corresponding external resource. This works seamlessly in both Tables and Views without requiring any special formatting. ### Supported Prefixes Synapse supports a curated subset of over 200 [http://bioregistry.io](http://bioregistry.io/) prefixes. You can view the complete list of Synapse-supported prefixes in the [Sage-Bionetworks bioregistry-collection repository](https://github.com/Sage-Bionetworks/bioregistry-collection). Some commonly used prefixes include: * **pubmed** - PubMed articles * **cbioportal** - [cBioportal.org](http://cbioportal.org/) datasets * **uniprot** - UniProt protein database * **geo** - Gene Expression Omnibus data series * **arrayexpress** - ArrayExpress datasets * **interpro** - InterPro protein families * **chebi** - Chemical Entities of Biological Interest * **go** - Gene Ontology terms ## Using Bioregistry Links in Tables To add external links to a Table: 1. Navigate to your Table and click **Table Tools** , then **Show Table Schema** 2. Click **Edit Schema** to modify your table structure 3. Add a new column or use an existing text column where you want to include external links 4. Set the **Column Type** to **String** for text-based identifiers 5. In your table data, enter values using the format: `prefix:identifier` For example, if you have a column for publications, you might enter: * `pubmed:12345678` * `doi:10.1038/nature12345` * `pubmed:98765432` These will automatically become clickable links when viewed in the table. ## Using Bioregistry Links in Views When you create a File View, Project View, or Submission View that includes files or tables annotated with bioregistry identifiers, those links will be preserved and clickable in the view. To include external links when creating annotations for files in a view: 1. Create or edit annotations on your files with keys that correspond to external databases 2. Use the bioregistry prefix format for the annotation values 3. Create a view that includes these annotated files 4. The bioregistry identifiers will appear as clickable links in the view ## Examples ### Linking to Publications In a research dataset table, you might have a "Publication" column with values like: * `pubmed:12345678` - Links to the PubMed article * `doi:10.1038/s41597-022-01807-3` - Links to the DOI resolver ### Linking to Protein Data In a proteomics study, you might reference: * `uniprot:P04637` - Links to the UniProt entry for the p53 protein * `interpro:IPR016380` - Links to the InterPro family page ### Linking to Chemical Compounds For metabolomics or drug studies: * `chebi:CHEBI:16991` - Links to the ChEBI entry for the compound * `pubchem:123456` - Links to the PubChem compound page ## Requesting New Prefixes If you need to link to an external database that isn't currently supported, you can request the addition of new bioregistry prefixes by visiting the [bioregistry-collection repository](https://github.com/Sage-Bionetworks/bioregistry-collection) and creating a new issue or pull request to request the prefix. ## Best Practices **Consistent Formatting** : Always use the exact prefix format as specified in [http://bioregistry.io](http://bioregistry.io/) . Prefixes are case-sensitive and should match exactly. **Documentation**: When sharing tables or views containing external links, consider documenting which external databases are referenced and what the identifiers represent. For additional help with Tables and Views, see [Organizing Data With Tables](https://help.synapse.org/docs/Organizing-Data-With-Tables.2011038095.html) and [Views](https://help.synapse.org/docs/Views.2011070739.html). *** ** * ** *** *The* [http://bioregistry.io](http://bioregistry.io/) *project is described in: Hoyt, C. T., et al. (2022). Unifying the identification of biomedical entities with the Bioregistry. Nature Scientific Data, 9, 714.* --- language: "en" --- # Links Synapse links are shortcuts to other data in Synapse. You can use links to create bookmarks for any file, table, view, folder, or project, rather than duplicating them in multiple folders. Links receive their own synIDs that are unique from the original linked item. ## Creating a Link via the Synapse UI Navigate to the file, table, folder or project that you wish to link. Click the **Tools** menu and select **Save Link**. ![image-20230308-191656.png](https://docs.synapse.org/__attachments/a_8a9d7c5c736ec1e7e5a4b4224d9a149897d922615054afca3e62a60152d576d4/image-20230308-191656.png?cb=e978f5801f56b2175610cabb5c46ed00) In the pop-up window, select a destination folder or project. This location is where the link will appear once you create it. Click **Create Link** to save. ![image-20230308-191949.png](https://docs.synapse.org/__attachments/a_86671a2375a9adabb34d54de7829de9eb6dc25dbd1e86a7358b4ba57c730d9bf/image-20230308-191949.png?cb=1fd3cace878bcb2ce0944f86135385a3) Once you've created a link, the final result will appear in your selected destination. To remove the link, click the red link icon. ![link-entity.png](https://docs.synapse.org/__attachments/a_557bb2f02c0cba988da90f68239d70e048eea9f5bff6b5cd644e11a4d467cda7/link-entity.png?cb=b48b47bcc876dffb614ac9e1ee51c487) ## Creating a Link Programatically **Python** import synapseclient syn = synapseclient.login() # Add a local file to an existing project (syn12345) on Synapse # targetId is the synapse id of the file, table, etc that you want to link # targetVersion is optional, if no version is defined, the link will always point to the newest version # parent is the folder or project where you want to link to exist linkEnt = synapseclient.Link(targetId="syn12345", targetVersion=1, parent="syn2345") linkEnt = syn.store(linkEnt) **R** library(synapser) synLogin() # Add a local file to an existing project (syn12345) on Synapse # targetId is the synapse id of the file, table, etc that you want to link # targetVersion is optional, if no version is defined, the link will always point to the newest version # parent is the folder or project where you want to link to exist linkEnt <- Link(targetId="syn12345", targetVersion=1, parent="syn2345") linkEnt <- synStore(linkEnt) --- language: "en" --- # Managing Custom Metadata at Scale This use case will combine concepts from [Annotating Data With Metadata](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md), [Views](https://docs.synapse.org/synapse-docs/views.md), [Uploading and Organizing Data Into Projects, Files, and Folders](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data-into-projects-files-and-folders.md). You will learn how to: * Create a manifest * Upload 100 files * Edit annotations on these files using the Synapse programmatic clients ## Annotation Dictionaries The Sage Bionetworks maintains [annotation dictionaries](https://github.com/sage-bionetworks/synapseAnnotations) in GitHub. You can use the terms in this repository as a starting point, or you can create your own annotation dictionary. ## Batch Upload Files with Annotations To batch upload files, create a tab-delimited manifest which contains, at minimum, the columns `path` and `parent`. You can also add additional annotations as columns in your manifest. For example, your manifest might have the following headers: `path`, `parent`, `specimenID`, `assay`, `species`, `platform`, `sex`, and `fileFormat`. * **path**: the local path to your file * **parent**: the Synapse ID (in the format syn123456) of the folder or project where your files will be uploaded * **specimenID**: the unique identifier for each of your specimens * **assay**: the technology used to generate the data in this file (for example, RNASeq, ChIPSeq, wholeGenomeSeq) * **species**: the species of your sample (for example, Mouse, Rat, Human, Triceratops) * **platform**: the hardware used to generate the data (for example, HiSeq2500, Affy6.0, HoodDNASequencer) * **sex**: a label assigned at birth based on biological attributes (for example, male or female) * **fileFormat**: is the type of file (e.g. fastq, R script) | path | parent | specimenID | assay | species | platform | sex | fileFormat | |-------------------------------------|--------|------------|----------------|---------------------------|------------------|--------|------------| | /local/path/to/velociraptor_b.fastq | syn123 | blue_1 | wholeGenomeSeq | Velociraptor mongoliensis | HoodDNASequencer | female | fastq | | /local/path/to/velociraptor_d.fastq | syn123 | delta_1 | wholeGenomeSeq | Velociraptor mongoliensis | HoodDNASequencer | female | fastq | **Save** this file in a tab-delimited format called `velociraptor_manifest.tsv`. Files can be uploaded all at once with a manifest file. If you would like to do a "dry run" validation of the file before uploading, you can add the parameter `dryRun = True` to the function `syncToSynapse`. Note that the `dryRun` feature checks everything, but does **not** upload the files. * validate the manifest and upload files in the [Python client](https://python-docs.synapse.org/build/html/synapseutils.html#synapseutils.sync.syncToSynapse). * validate the manifest and upload files in the [R client](https://github.com/Sage-Bionetworks/synapserutils#batch-process). ## Create a File View (web) Once the files have been uploaded with annotations, you can use a file view to query, facet, and bulk manipulate the files and metadata. To create your file view: 1. Navigate to your project. 2. Go to the tables tab, select **Tables Tools** in the upper right corner, and click **Add File View**. 3. In the resulting pop-up, give the new file view a name. 4. Select the container (Synapse project or folder) of files, and click **Next** . In this case, you would want the synID of the `parent` column in the manifest. 5. Select the columns you would like to keep. Since we are going to edit the annotations later, make sure you have the column `etag` listed as one of your columns. 6. **Add All Annotations** at the end of the opened window will add all existing annotations. 7. Click **Finish** to create the file view. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on file views, see [Views](https://docs.synapse.org/synapse-docs/views.md). ## Perform a One-Time Annotation Update or Deletion (web) An annotation for a single file can be modified in the web client view. For example, you can update`specimenID`:`delta_1` to `specimenID`:`echo_1`. 1. From your view, select the **pencil icon** to edit query results. 2. In the pop-up window, find the single value you want to change and edit the field. 3. Scroll down to the bottom and click **Save** to update the file view. ## Perform a Bulk Annotation Update or Deletion A bulk annotation update is required in the case that `species`:`Velociraptor mongoliensis` should be modified to `Utahraptor ostrommaysorum` in all 100 files. **Web** To download the annotation values from the web client: 1. Navigate to the your file view. 2. Click the **Download Options** button to the right of the query bar of the file view. 3. Click **Export Table** to download the file view. Be sure to have `Include row metadata (Row ID and Row Version)` selected when downloading. Now that you have the file view downloaded, you can edit the values using your preferred tool (Python, R, Excel, or other). With the changes saved, go back to the file view in your browser. 1. Click **View Tools** located at the upper right of the file view. 2. Select **Upload Data to View** from the dropdown. 3. Browse and choose the edited file. 4. Click **Next** to preview the uploaded file. 5. Click **Update Table** to update the file view and populate the changes to all the files in Synapse. **Programmatic clients** Alternatively, download the file view with the R or Python client, then: 1. Query for the file view with `synTableQuery()` or `syn.tableQuery()`. To delete all the annotations of a key, you have to keep the column in the file view but remove the values. 2. Update and then store the annotations in the [R client](https://r-docs.synapse.org/articles/views.html#updating-annotations-using-view) or [Python client](https://python-docs.synapse.org/build/html/Views.html#updating-annotations-using-view). --- language: "en" --- # Managing Data Access With Teams Not familiar with Synapse teams? First, learn all about them [here](https://docs.synapse.org/synapse-docs/teams). For complex Synapse projects, administrators manage a wide variety of data that requires different permissions for different user groups. In this scenario, creating a system for managing data access with Synapse teams is recommended. This method allows you to group users together according to the level of access they require and then assign permissions to the entire team rather than to many individuals. When creating teams to manage data access, consider the following questions: * Who needs to be able to view this project? * Who needs to be able to edit or add content to this project? * Who should be in charge of changing permissions to modify user access? * Does any content in this project need different permissions than the whole project? For example: * Raw data vs. processed data folders shared with specific groups * Internal meeting notes in a private folder vs. methodology and SOP documents shared publicly ## Using Teams for Permissions Teams are groups of Synapse users; learn more about [creating and managing teams](https://docs.synapse.org/synapse-docs/teams#Creating-a-Team). If you are working with many Synapse users on a project, and you want to allow some users to view or download data, and other users to add new data, you should consider using teams to manage your project permissions. For example, you can create a "project administrators" team and grant that team permission to administer the project. Then you can add or remove people from the team rather than modifying the sharing settings on the entire project to grant each individual administrative permissions. This approach is especially useful if you have more than one project that the same group of people will be working on. Using teams for permissions can also help prevent administrative errors like forgetting to remove someone from a project if they leave your collaboration. ## Recommended Team Types For many collaborations, a small group of users administers the project, a larger group contributes data, and an even larger group downloads that data for their independent research. In these cases, we recommend creating groups for each of these permission types: an administrative team with "administer" permissions, a data curation or content creation team with "can edit" permissions, and a downloading team with "can view/download" permissions. Because permissions are additive, a user who is in all three teams has the permissions of the highest level granted. In other words, if you add a single user to three different teams, "administer", "can edit", and "download", the user will have "administer" permissions. If you remove that user from the "administer" team, they will have "can edit" permissions. ## Local Sharing Settings Sometimes, users wish to create private spaces for certain groups within larger, public projects. This is possible using local sharing settings to restrict content to specific teams. To create a private folder within a public project, click the **Project Settings** button, then select **Project Sharing Settings** . Add specific team names that should have access to all project content. Next, click **Make Public** and select the appropriate access levels for all registered users and anyone on the web. Next, navigate to the new Folder, and click on **Folder Tools** . Select **Folder Sharing Settings** , then click **Create Local Sharing Settings** . Click on the **Make Private** button and confirm that fields for **All registered Synapse users** and **Anyone on the web** have been removed.**Save** your changes. The folder can then be shared only with the specific teams that the entire project is shared with, and not the general public. Removing local sharing settings on an item will assign default permissions from the parent folder or project. ## Triaging Sharing Settings with Views Creating local sharing settings for many folders, sub-folders, or other items in Synapse can become complex to manage as a project grows. You or another administrator may alter local sharing settings unintentionally, or you may want to audit your sharing settings periodically. One way to manage these settings is by [creating a view](https://docs.synapse.org/synapse-docs/views) to see sharing settings at a glance. An important component to managing sharing settings is the benefactor ID, or `benefactorId`. This identifier is the name of the parent folder or project that sharing settings are inherited from. When you first create a project, the project itself is the "benefactor" for sharing settings, meaning all items within that project inherit the same settings, and there is only one benefactor ID for everything in the project. When you create local sharing settings on an item, a second benefactor ID is created. The project is still providing the sharing settings for the rest of the content, but wherever you set local sharing settings on a folder, this folder is now the benefactor for anything inside it. The benefactor ID is useful because you can include it in a view of every item in the project and use it to review all permissions at once. This view will help you find items where local sharing settings are active, causing new benefactor IDs to appear. In the example below, the folder name, benefactor ID, and the project ID for three folders are shown in a view. ![View-benefactorID.png](https://docs.synapse.org/__attachments/a_a2d158fecb055b38c45885288e0b7a968cb6c1d4bc46309cb0c144c656ffcfce/View-benefactorID.png?cb=9008f9765234f1c268be2e01df63038e) All of the folders in this view belong to the same project,` Demo_Project`, and their corresponding project IDs in the third column are the same. In the second column, `Folder 1` and` Folder 3` are both inheriting their sharing settings from the parent project, `Demo_Project`. However, Folder 2 has local sharing settings applied that are different from the parent project, and the benefactor ID has changed. For views with hundreds or thousands of rows, you can also use a SQL-like query to identify items that have different permissions than the parent project. The query below compares the benefactor ID of each item to the project ID and displays any items where the two do not match. You can modify the query to look at at files, tables, or all content in your view: SELECT id,parentId,type FROM syn12162270 WHERE type = 'folder' AND benefactorId <> projectId For more information on how to use SQL-like queries in Synapse, see [Querying Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/versioning-tables-views-and-datasets#Querying-a-Version). --- language: "en" --- # Managing Files and Folders This tutorial will guide you through fundamental Synapse features for managing files and folders with the web interface. You will learn how to: * Add a folder to a project and upload a file * Annotate a file * Add a description to your folder with a wiki * Download a file ## Prerequisites Anyone can browse public content on the Synapse website, but to download and create content using this tutorial, you will need to [register for an account](https://www.synapse.org/register) using your email address. You will receive an email message for verification to complete the registration process. To upload files to Synapse, you will need to perform the additional step of becoming a [certified user](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md). Because Synapse stores data from human subjects research, we require that you demonstrate an understanding of our governance, including comprehensive privacy and security issues. You can complete your certification by taking a short [certification quiz](https://www.synapse.org/#!Quiz:Certification) on Synapse. Before you begin this tutorial, make sure you have already created a project. If you haven't already, see the use case [Setting Up a Project](https://docs.synapse.org/synapse-docs/setting-up-a-project.md). ## Uploading a File Synapse can be used to store files of all types. All files that you upload to a project will appear in the **Files** tab, which is the root directory. You can also create nested folders within a Synapse project to manage files and keep them organized. To upload a file: 1. Within a project, navigate to the **Files** tab. 2. Click the **File Tools** menu to select **Upload or Link to a File**. 3. Use the **Browse** button to select the file, or drag and drop it to upload, and click **Save**. To create a folder and upload a file: 1. Navigate to the **Files** tab. 2. Use the **Files Tools** menu to select **Add New Folder**. 3. Decide on a folder name and click **Save**. 4. Navigate into your new folder by clicking on the name, then use the **Folder Tools** menu to select **Upload or Link to a File**. 5. Use the **Browse** button to select the file, or drag and drop it to upload, and click **Save**. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn about setting restrictions on your data, see [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Other features available for files and folders include [Digital Object Identifiers (DOIs)](https://docs.synapse.org/synapse-docs/digital-object-identifiers-dois.md), [Versioning Files](https://docs.synapse.org/synapse-docs/versioning-files.md), and [Provenance](https://docs.synapse.org/synapse-docs/provenance.md). ## Adding a Folder Wiki You can add a description to the folder you just created by adding a folder wiki. Wikis can be added to projects, folders, tables, and files in Synapse, and they can help you communicate essential details, instructions, or other information about your work to other collaborators. To add a wiki to the folder you just created: 1. Navigate to the **Files**tab. 2. Select the folder you just created and navigate into it. 3. From the **Folder Tools** menu, select **Edit Folder Wiki**. 4. Type a description of your folder contents in the wiki editing window, then click **Save**. Your wiki content will be visible to anyone who has access to your folder. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more, see [Creating and Managing Wikis](https://docs.synapse.org/synapse-docs/creating-and-managing-wikis.md). ## Annotating Files Annotations help you find files and search for data, and they are an essential component of the [FAIR principles](https://www.go-fair.org/fair-principles/) for data management. In Synapse, annotations are stored as key-value pairs, where the key defines a particular aspect of your data (for example, species, assay, file format) and the value defines a variable that belongs to that category (mouse, RNAseq, .bam). You can add annotations to projects, files, folders, tables, views, and datasets in Synapse. In this example, you will annotate the file that you uploaded in the previous section of this tutorial: 1. Navigate to the file and click on the file name. 2. Click on the **File Tools** menu and select **Annotations**. 3. A new panel will appear with a list of any previously added annotations for this file. Click **Edit**to add new annotations. 4. In the pop-up window, add your annotations one at a time. Use the **+** icon to add multiple values for a single key and the **x** icon to remove values. Use the **Add New Key** button to add a new key. ![edit_annotations.png](https://docs.synapse.org/__attachments/a_277a1b7a534fc7056f30ace7cfc067a3ce2145ac092f6f0e8c806a451e33fef6/edit_annotations.png?cb=f0420b3a43dbe82c0b0ac6aea2f36d91) Once you add annotations to a file, you can use them to [query for data](https://docs.synapse.org/synapse-docs/querying-tables-views-and-datasets.md), or you can build [custom views](https://docs.synapse.org/synapse-docs/views.md) to display annotation information for many files at once. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on creating annotations, see [Annotating Data With Metadata](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata.md). ## Downloading Files You can download files from the Synapse UI or using one of the programmatic clients. If you are downloading files from the web, you can either download single files at once, or you can add multiple files to a download list, which is similar to an online shopping cart. On the web, the maximum size of a download is 5 GB, or a maximum of 100 files if using the download list. To download larger files in greater numbers, we recommend using one of the [programmatic clients](https://docs.synapse.org/synapse-docs/api-clients-and-documentation.md). Before you can download a file from Synapse, you must determine whether you have access to it. Click on a file name and look for a green or yellow lock symbol. A green unlock symbol means that the file is available for you to download. A yellow locked symbol means that the file is not available to you. Contact the project owner or click **Request Access**. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more about data access, see [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). ![request_access.png](https://docs.synapse.org/__attachments/a_823207dfd5d3c11cc8633517b472668cc0bfd1cff7a57d3e629d36e3b7ca938b/request_access.png?cb=cf2b636c899fa3c371b8fe27a526b899) To download a file from the web: 1. Navigate to the **Files** tab of your project and find a file to download. Note that you cannot download entire folders from the web interface, so you may need to expand a folder in the directory to see its contents. 2. Click the file name to see a detail page. 3. Click **Download Options** button and then **Download File**. 4. From this menu, you can also choose to **Add to Download List** , which will start a list of files that you can download all at once. Alternatively, you can view the **Programmatic Options** to see example code that you can use to download files with one of the programmatic clients. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on downloading from the web, see [Downloading Data From the Synapse UI](https://docs.synapse.org/synapse-docs/downloading-data-from-the-synapse-ui.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For information on downloading data using an API client, see [Downloading Data Programmatically](https://docs.synapse.org/synapse-docs/downloading-data-programmatically.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on installing and using our API Clients, see [API Clients and Documentation](https://docs.synapse.org/synapse-docs/api-clients-and-documentation.md). --- language: "en" --- # Managing Metadata with Curator 🆕 Curator webinar slides are available [here](https://drive.google.com/file/d/1Qus9B8I-DnheVg8gmOK5E-1XIROl-k9L/view). ## About Curator is a spreadsheet-style tool for adding metadata to files and entering structured research information, including details about datasets, assays, participants, biospecimens, research tools, and other project-specific information. It is currently available through the Synapse platform. Users can enter information manually or work with Curie, Curator's AI chat assistant. Curie can explain metadata requirements, suggest appropriate values, and help populate fields based on the project context. Users can then review, edit, and approve all suggested changes before they are applied. Curator supports two main workflows: * **File annotations:** Metadata is attached directly to individual Synapse files as annotations. * **Structured records:** Research information is stored in a single spreadsheet, with each row representing one record. The main difference between these workflows is where and how the information is stored. ### File annotations Annotations are key-value pairs that describe important details about a file. For example, a sequencing file might include annotations such as `assayType: RNA-seq`, `organism: Homo sapiens`, and `tissue: blood`. If you have used Synapse before, you may recognize annotations by the tag icon shown here ![annotation.png](https://docs.synapse.org/__attachments/a_ad9f5f2161446d45dd458a67528cc3b7937cdaa0ce4b1d0f038e2379bf630fee/annotation.png?cb=a7d551ca8f0234adbe8ce7fc2815b260) . Annotations are used by downstream systems, such as Synapse data portals, to support search, filtering, and data discovery. This is why accurate and complete annotations are important. Synapse annotations are publicly visible when the sharing settings are set to **"Anyone on the web"** with **"Can view"** access. Do not include private or sensitive information in annotations. When adding file metadata in Curator, the two leftmost columns are **name** , which displays the filename, and **id** , which displays the file's Synapse ID (e.g. `syn12345678`). Columns to the right of **name** and **id** capture additional metadata defined by the associated metadata template and are displayed as the header row. The table below shows an example for RNA sequencing FASTQ files, annotated with metadata fields such as *assay* , *tissue* , *species* , *file_type* , and *sample_id*. | ***name*** | ***id*** | **assay** | **tissue** | **species** | **file_type** | **sample_id** | |---------------------|-------------|-----------|------------|-------------|---------------|---------------| | sample1_read1.fastq | syn12345678 | RNA-seq | blood | human | fastq | 001 | | sample1_read2.fastq | syn12345679 | RNA-seq | blood | human | fastq | 001 | | sample2_read1.fastq | syn12345680 | RNA-seq | blood | human | fastq | 002 | | sample2_read2.fastq | syn12345681 | RNA-seq | blood | human | fastq | 002 | ### Structured Records Structured records are used to capture research information about datasets, assays, participants, biospecimens, research tools, and other project-specific information. This information is stored in a single spreadsheet-like record set. Each row represents one record, such as one participant, sample, specimen, dataset, or assay. #### Definitions **Record set** A table containing a collection of related records that describe the same type of research information. For example, a record set may contain participants, samples, specimens, datasets, or assays. Record sets can be uploaded from tabular files such as CSV or TSV files. **Record** A single row in a record set. Each row represents one item, such as one participant, sample, specimen, dataset, or assay. **Prmary Key** A user-defined value that uniquely identifies a record within the record set. Examples include `participant_id`, `sample_id`, or `specimen_id`. #### Uploading and Updating Records Columns marked with a key icon are **primary key columns**. Be sure to complete these columns for every row. Synapse uses the primary key values to identify each record and determine whether your upload should add a new record or update an existing one. When you upload a record set: * A row with a new primary key value is added as a new record. * A row with a primary key value that already exists updates the matching record. * Existing records that are not included in your upload remain unchanged. Some record sets use more than one column as a combined primary key. In these cases, complete all primary key columns so Synapse can correctly identify and update the record. As shown in the table below, record metadata capture additional clinical context (for example, diagnosis, age bracket, and sex) for the files listed above. Record metadata are stored in a separate table so that multiple files can be linked to the same participant or sample without repeating the same information. This keeps metadata consistent and easier to update. | **sample_id** | **diagnosis** | **age_bracket** | **sex** | |---------------|---------------|-----------------|---------| | 001 | Glioblastoma | 50--59 | Male | | 002 | Glioblastoma | 40--49 | Female | Whether metadata is applied as file or record metadata depends on the type of content. In general, when you upload files, you will always have at least file metadata applied. In some cases, you may also have record metadata that describes patients, biospecimens, etc. 💡 [**Test your knowledge!**](https://sagebionetworks.jira.com/wiki/spaces/DRAFT/pages/4461527055/Test+your+knowledge+on+Record+vs.+File+based+metadata) This diagram provides an overview of the metadata contribution workflow in Curator. Additional details and instructions for each step are included in this documentation. ![Curator_Workflow.png](https://docs.synapse.org/__attachments/a_b4f9511056fb00c76dc0c1e237835b222629d65275fe5bfb7f786448054376e1/Curator_Workflow.png?cb=691816a4c6fdc9655b825b6c8dc238eb) ## Requirements for using Curator To use Curator, you must: * [Create a Synapse account](https://www.synapse.org/register) * Complete the [Synapse Certification Quiz](https://www.synapse.org/#!Quiz:Certification) * Have **View**access (or higher) to the Synapse project, granted by the project Administrator * * Have **Edit**access (or higher) to any files you will work with, granted by the project Administrator ## Setting up Curator ### **Data Coordination Plan (i.e. Data Coordinating Centers)** If you are part of a **Data Coordination plan** (i.e. data coordinating center such as ADKP, ARK, ALS, ELITE, Classic, HTAN, MC2, or NF-OSI) your project setup is handled by **Sage Bionetworks**. Curator is enabled automatically on your behalf, and no additional configuration is required from you. If you have questions or need support, please contact your consortium's service desk: * [AD Knowledge Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12) * [ALS Portal Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [ARK Portal Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/11) * [Classic Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [ELITE Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [HTAN Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/1) * [MC2 Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/10) * [NF-OSI Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/2) * [NAMHub Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/89) If none of the above apply, please submit questions to the [Synapse Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9) ### **Free Basic Plan** If you manage your own project on the free Basic plan, Curator is disabled by default and hidden in the user interface. To enable it, follow the python client instructions: . ## Navigating to Curator There are two ways to navigate to Curator: 1. **Curator Dashboard** : View curation tasks assigned to you across all Synapse projects. ![image-20260608-121411.png](https://docs.synapse.org/__attachments/a_1bade0f4f2742e8a8090c79cfe1b8651860be8a113d969869e502a69a1bc63fb/image-20260608-121411.png?cb=7b0ef6d1c5152e47cd95787758281f99) 2. **Tasks and Actions tab:** Within a Synapse project, select the Tasks and Actions tab to view and manage all tasks associated with that project. ![image-20260802-141752.png](https://docs.synapse.org/__attachments/a_5684eab6ec0fe6bd94a363919a71ac3e977858f3fe4471c3081765bb38caff52/image-20260802-141752.png?cb=6766a5b72ca524fea321e227f16c7227) ### What Are Tasks? Tasks guide you through the steps needed to complete required or recommended work for your project or data. ### How Curation Tasks Relate to Your Project Files After you upload data, use the Tasks and Actions tab to see what metadata or curation work still needs to be completed. The tasks you see depend on the data in your project. Here is an example of 4 types of data and 4 tasks you might see in the Tasks and Actions tab. ![image-20260519-211356.png](https://docs.synapse.org/__attachments/a_a9a3b7b886aea8433b6c4f61a9e991907bc5c1faa3655ff3186d7f7fbd8b8bd9/image-20260519-211356.png?cb=b09d5450d47416e4329b952e7259a85d) **If you are unsure which task to complete, or if you do not see an expected task, please contact your data manager or project administrator for guidance.** ## Permissions Synapse permissions are grouped into five levels: **View, Download, Edit, Edit and Delete, and Administrator.** To view the Tasks and Actions tab, a user must have at least **View** permission on the Synapse project. To **open curator** , the user must have at least **Edit** permission on **every file**that needs annotations. **How to check or change sharing settings** : For a **project** , go to the top-right corner and select **Project Tools → Sharing Settings** . **Folders** and **files** also have their own sharing settings. Navigate to the specific folder or file, then open its tools menu and select **Sharing Settings**. ## Collaboration Q. How do I make a grid session collaborative? 1. Click the **Edit**button next to the task ![image-20260608-132620.png](https://docs.synapse.org/__attachments/a_9ab29d986f6c32fd703556d439844b3421995c8a201f112bbb2924865dcce039/image-20260608-132620.png?cb=3e3aba7047fd0ec87d59485d0f453902) 2. Select one of the following authorization modes: **Share with assignees only** Access to the grid session is limited to the user or team listed in the Assignee field. Assignees must also have edit access to all files included in the curation task. **Share with all editors** Anyone with edit access to all files included in the curation task can access the grid session. Once collaboration mode is enabled, link sharing becomes available. Copy and paste the URL to share the grid session with other editors. To see who changed a cell, hover over the arrow in the upper-right corner of the cell. ![image-20260608-134757.png](https://docs.synapse.org/__attachments/a_554a584ef519463076bf14738a907b6c0ab3be23eb6409d816c81d7743e50c48/image-20260608-134757.png?cb=ca90fa745992066b6b1c9618bdeecfff) ## Assignee The Assignee field helps organize work by assigning a task to a specific user or team, making it easier to filter tasks and track ownership. If the collaboration mode is set to "Share with assignees only," the Assignee field also determines who can access the grid session. Keep in mind that assignees must also have edit access to all files included in the task. **How to change the assignee:** To change the Assignee field in the Tasks and Actions tab, a user must have at least **Edit** permission on the **Synapse project**. Hover over the Assignee and click the pencil icon ( :pencil_icon: ). Toggle "View only tasks assigned to me" to show only tasks assigned directly to you or to a team you belong to. ![image-20260802-141909.png](https://docs.synapse.org/__attachments/a_cafc79ffb1c36dc23ab74a0db6f9f7e0da79c18564dbf326d23abfc1da5c6661/image-20260802-141909.png?cb=55d4d7efea3a083c54638df62d316829) ## Working in Curator 1. From the **Tasks and Actions tab** , click **Open Curator** . ![image-20260802-142134.png](https://docs.synapse.org/__attachments/a_8ac783dd005c9dbb7ab019052f2ee1c372e11e3b39ca1103a0d0c85e3e40af2b/image-20260802-142134.png?cb=50c3b699b12ab52537c3a7531d124ba5) For file metadata, the grid displays the list of files in the leftmost column. For record-set data, the grid displays all records in the dataset, either populated from a previous session or with empty cells if no metadata has been entered yet. Column headers indicate the fields or properties that need to be completed. * **Column with \***= required fields * **Red cells** = missing or invalid values * **White cells** = valid values **Dependencies (conditional fields)** Some fields are conditionally required based on values entered in other fields, as defined by the schema. When a dependency is triggered: * A column header may add an \* to indicate it has become required. * Cells in the affected column may change from **white to red** if values are missing or no longer valid under the new condition. ![rnaseq_view_feature_labels_v3-20260802-175329.png](/__attachments/a_fc7ff5af345c1466b17e78abb86644282972017f798f6d9cabcb7229ba623916/rnaseq_view_feature_labels_v3-20260802-175329.png?cb=e55249b1d0ed809e39e3ab7ebcc0323c) **Grid actions** * **Add rows:** Select **+ Add** to create a new row. * **Edit faster:** Right-click a cell (or **Control + Click**) to open the context menu: * Copy * Cut * Paste * Insert row below * Duplicate row * Delete row * **See more fields:** Scroll horizontally to view additional columns. * **Bulk upload:** Upload a **CSV** to populate the grid with metadata *(record submissions only).* * **View JSON Schema**at the top of the page to review column details. 4. Enter metadata by clicking directly into a cell and typing a value. For fields with a controlled vocabulary, click the cell and select one or more values from the drop-down list. 5. Click **Sync Changes** for file metadata or **Apply Changes** button for record metadata. 6. Once applied, your updates take effect immediately. For file metadata, these become Synapse annotations and become the project's Synapse annotations. **💡 TIP: Try out keyboard shortcuts to move through the table faster.** |------------------------|----------------------|---------------------| | **Action** | **Windows shortcut** | **Mac shortcut** | | Copy | Ctrl + C | Command + C | | Cut | Ctrl + X | Command + X | | Paste | Ctrl + V | Command + V | | Insert row below | Enter | Return | | Duplicate row | Ctrl + D | Command + D | | Go to first row | Ctrl + ↑ | Command + ↑ | | Go to leftmost column | Ctrl + ← | Command + ← | | Go to rightmost column | Ctrl + → | Command + → | | Go to last row | Ctrl + ↓ | Command + ↓ | | Undo | Ctrl + Z | Command + Z | | Redo | Ctrl + Y | Command + Shift + Z | | Delete | Delete | Delete | ## Uploading a CSV For record metadata, you can fill out your data in another tool (like Excel or Google Sheets) by downloading a CSV template and uploading it back into the grid. 1. Go to the **Tasks and Actions** tab. 2. Click **Open Curator** next to the relevant task to open Curator. 3. In Curator, click **Download** to export the spreadsheet as a CSV. 4. Open the CSV in your preferred tool and complete it offline. **Important:** Don't change the column headers. 5. Back in Curator, click **Upload** and select your completed CSV file. ## How uploads create and update records Every row must include a **primary key** column. Curator uses this ID to decide whether each row should create a new record or update an existing one. When you upload a record set: * **New IDs** (not already in the record set) are added as **new records**. * **Existing IDs** overwrite (update) the **record with the same ID**. * **Records not included** in the uploaded file are **left unchanged**. ## Curie, the AI Grid Assistant Curie is an AI-powered chat assistant designed to help you curate, clean, validate, and populate tabular data directly within a grid. It understands the grid's schema and can analyze selected rows or the entire dataset to provide guidance, corrections, and transformations. Curie is especially useful when working with structured data that must conform to specific validation rules or when you want to quickly populate a grid using a project description or other large blocks of text. ### Accessing Curie: To open Curie: 1. Navigate to the grid. 2. Click the **Open Chat** button in the upper-right corner of the grid. The chat panel will open, allowing you to ask questions or give instructions related to the grid. 3. To close the chat, click the **×** in the upper-right corner of the chat panel. Closing the chat preserves your current conversation. 4. If you leave the page or refresh your browser, the chat session will end. A new chat session will start when you reopen Curie 5. Curie can make mistakes, so please check all work prior to submitting! ### What Curie Can Help With * Identify errors, missing values, and duplicate records in the grid. * Normalize and clean inconsistent or poorly formatted data. * Explore, filter, and summarize data in the grid. * Create new columns or derive values from existing data. **Example Prompts:** * *Can you check the selected rows for schema validation errors and explain what's wrong?* * *These rows have inconsistent date formats in the sample_date column - can you fix this?* * *I'm seeing errors in the species column for this data. What's the issue, and can you fix the selected rows?* * *For the selected rows, can you generate a sample_id by combining project_id and specimen_number?* * *Are there any common issues across the selected rows that I should address before submission?* ### 💡 Tips * Be specific about which rows or columns you want Curie to work on. * Use prompt templates for repeatable workflows. * Review suggested changes before applying them to ensure correctness. ### What Curie can see: * Content currently visible to you in the grid, and * Information you choose to provide in the chat. It follows the same permissions and access controls as the rest of Synapse and cannot see anything you do not already have access to. The Grid Agent is hosted in **AWS Bedrock** and does not use submitted content for model training. As with any Synapse feature, please use care when working with sensitive information and follow the [Synapse Terms of Service](https://www.synapse.org/TrustCenter:TermsOfService). ## Metadata Requirements ### Consortium or Funder Requirements If you are part of a larger consortium or are working under a specific funder, metadata requirements are often already defined for you. These requirements ensure consistency across datasets and help make data interoperable within a research domain. You do **not** need to configure or apply these requirements yourself. When you fill out the Curator grid, the appropriate metadata schema is automatically applied for you. It can be helpful to be aware of these metadata requirements ahead of time, so you know what information to collect during data generation and preparation. Below are examples of consortium-managed metadata dictionaries and data models that Curator uses behind the scenes: * [AMP-AD Metadata Dictionary](https://sagebio.shinyapps.io/amp-ad-metadata-dictionary/) * [ARK Portal Data Model Dictionary](https://ark-portal.github.io/data_model/) * [ELITE Metadata Dictionary](https://eliteportal.github.io/data-dictionary/) * [HTAN Metadata Dictionary](https://docs.humantumoratlas.org/data_model/overview/) * [MC2 Center Data Models Explorer](https://mc2-center.github.io/data-models/) * [NF Metadata Dictionary](https://nf-osi.github.io/nf-metadata-dictionary/) ### Individual Synapse Users If you are an individual Synapse user and are not part of a larger consortium, we plan to make this easier in the future by offering a set of standard metadata schemas that you can choose from. These schemas will cover common data types and research workflows and will help guide you in providing clear, consistent metadata when uploading your data. For now, if you're interested in learning more or working with a more advanced setup, you can explore these resources: * [**Curator MVP setup via Python**](https://python-docs.synapse.org/en/latest/guides/extensions/curator/metadata_curation/), for defining custom metadata structures * [JSON Schemas](https://docs.synapse.org/synapse-docs/json-schemas.md) to see how metadata fields and validation rules are defined behind the scenes ## Validating The grid editor checks your entries as you type and points out missing or incorrect information. Highlighted cells and messages show you what needs attention. You can still click **Sync Changes** even if there are messages. This is on purpose, so you can save your work when some information isn't available yet (for example, if you don't know a required value or don't see the right option in the list). ### Where to See Validation Errors * **Row Indicator (Left Panel):** A highlighted row indicator means at least one value in that row is missing or invalid. * **Highlighted Cells:** Highlighted cells show exactly which values are missing, invalid, or incorrectly formatted. * **Validation error panel:** A panel at the top of the page lists any validation errors found in the metadata grid. Select an error message to jump directly to the affected cell or row. In this example, the validation panel displays 16 metadata errors, organized by row, column, or message. The highlighted error indicates that the **nSamples** field is missing a required integer value in **Row 1.** ![image-20260519-215851.png](https://docs.synapse.org/__attachments/a_894941bf7dd301c59522ad7d3f3b9cdf0504ea3e6c3db9fda92512356016e93f/image-20260519-215851.png?cb=608c8129184fa495aef06b9a968d81c1) ### **Resolving Validation Errors** | Message | Explanation | How to fix | |-----------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------| | **null is not a valid enum** | One or more **required fields** in the selected row are missing (null) or left empty, and the field expects a value from a predefined list. | Populate all required fields (blue columns) with a valid value from the allowed list. | | ``**is not a valid enum value** | The entered value does not match one of the allowed values defined in the metadata dictionary. | Select a valid option from the predefined list. If the correct term is missing, notify your DCC contact. | | **Required field** ``**is missing** | A required column defined in the schema is missing entirely from the submission. | Add the missing column and populate it with valid values for all rows. | | **Cannot convert** ``**to integer** | The value entered cannot be interpreted as a whole number. | Enter a whole number (e.g., `1`, `10`). Do not use text or decimals. | | **Value** ``**is not a whole number** | A decimal or non-integer number was entered where an integer is required. | Replace the value with a whole number. | | **Cannot convert** ``**to float** | The value entered cannot be interpreted as a numeric (decimal) value. | Enter a numeric value (e.g., `3.14`, `10`). | | **Cannot convert** ``**to boolean** | The value does not match accepted boolean values. | Use one of the supported values: `true`, `false`, `yes`, `no`, `1`, or `0`. | | **String length is less than minimum** | The entered text is shorter than the minimum allowed length. | Enter a longer value that meets the minimum length requirement. | | **String length is greater than maximum** | The entered text exceeds the maximum allowed length. | Shorten the value to meet the maximum length requirement. | | **String does not match pattern** | The value does not match the required format (pattern). | Update the value to follow the required format (for example, a specific ID or naming pattern). | | **Value is less than minimum** | The numeric value is below the allowed minimum. | Enter a number that meets or exceeds the minimum value. | | **Value is greater than maximum** | The numeric value exceeds the allowed maximum. | Enter a number that is less than or equal to the maximum value. | | **Expected type:** ``, **found:** `` | The value entered is the wrong data type for this column (for example, text entered where a number is required). | Replace the value with the correct type expected by the column (for example, enter a number instead of text). | | **\[null\] is not a valid date. Expected \[yyyy-MM-dd\]** | The date field is empty or not entered in the required format. | Enter a valid date using the required format: YYYY-MM-DD (for example, 2024-03-15). | ## Saving Edits are saved in real time as you enter information in the form. If you leave the page and return later, your information will still be saved. However, these changes are not committed to Synapse as final annotations until you click **Sync Changes** or **Apply Changes** If the update is successful, a green bar at the bottom of the screen will confirm that the sheet has been successfully updated. ![image-20260303-191958.png](https://docs.synapse.org/__attachments/a_cea2821a0515aa9682307ba82c3a69585f20642b76baa54db89d9988075c6167/image-20260303-191958.png?cb=ae9fbb69f6309c09a95cc77b324112ad) ### **Viewing File Metadata** Submitted file metadata for public projects can be viewed by **anyone on the web**. 1. Open the **Files** tab in the Synapse project. 2. From the folder tree, navigate to the folder associated with the records. 3. Hover over the **tag icon**:ann: next to an individual file name to view its annotations. * **A yellow tag** :y:**indicates that one or more annotations have invalid metadata.** 4. To see a complete list, click the file entity and click the :annotation: icon on the top right of the entity page. ### **Viewing Record Metadata** 1. Navigate to the **Files** tab in the Synapse project 2. Click the appropriate **folder** associated with the records. 3. Select the **record set entity** :re: , which will open a preview displaying the associated metadata. Click **Download File** to download as a csv. Please note: **Record data may be controlled access and is dependent on the entity sharing settings.** ![image-20260419-174629.png](https://docs.synapse.org/__attachments/a_1353f726beb4cba172fa3c744e26104cc2064d30800182b1b448aeb73360f088/image-20260419-174629.png?cb=3a43048b368394f809132b50d38f6270) ## Troubleshooting | Error message | How to fix | |-------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| | **The CSV file cannot be empty** | Ensure the CSV file contains at least one row. If a header is expected, include a header row followed by data. | | **Expected the first line to be the header but was empty** | Make sure the CSV includes a header row as the first line and that it is not blank. | | **The CSV header does not match the schema size** | Confirm that the number of columns in the CSV header exactly matches the expected schema. Do not add or remove columns. | | **The CSV header column "\" does not match the schema column "\" at index \** | Ensure the column names **and their order** exactly match the schema or **Curator**grid. Column order matters. | | **Unexpected processing error** | Retry the operation. If the error persists, contact support and include the CSV file and details about what you were attempting to upload. | ## Report a Bug If you run into an issue, have a question, or would like to request a new feature for **Curator** , please submit a support ticket to the [Synapse Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9) **To help us resolve your issue faster, please use the following format.** 1. **Synapse Project ID** (e.g. syn12345678) Provide the Synapse Project ID associated with the issue. 2. **Data Folder Syn ID**(e.g. syn12345678) Include the Synapse ID for the relevant data folder. 3. **URL of Curator Session**(e.g. https://synapse.org/Grid:default?sessionId=MTE4NTQ1OA%3D%3D\&taskId=1234) 4. **Issue Being Experienced** Clearly describe the problem, including what happened and what you expected to happen. 5. **Steps to Reproduce the Bug**List the exact steps needed to reproduce the issue so that the team can investigate it reliably. 6. **Supporting Evidence**Attach a video recording or screenshots that show the issue. Visual evidence is strongly encouraged and can help speed up troubleshooting. For questions related to specific projects or **Data Coordination Plans**, please contact your consortium's service desk instead: * [AD Knowledge Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12) * [ALS Portal Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [ARK Portal Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/11) * [Classic Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [ELITE Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [HTAN Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/1) * [MC2 Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/10) * [NF-OSI Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/2) * [NAMHub Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/89) ## Frequently Asked Questions ### **What features are available today?** Below is a summary of features available today and under construction. | Feature | Status | Notes | |---------------------------------------------|---------------------------------|-------------------------------------------------------------------------------------------------------------------------------| | Spreadsheet-style grid editor | ✅ Available | Row-and-column interface for metadata entry on Synapse | | File metadata (Synapse annotations) | ✅ Available | Metadata attached directly to files | | Record metadata (record sets) | ✅ Available | Separate tables for participants, samples, etc., with primary keys | | Project-level curation tasks | ✅ Available | Shared tasks visible to all collaborators | | Required versus optional field indicators | ✅ Available | \* = required no asterisk = optional | | Real-time validation while editing | ✅ Available | Highlights missing/invalid values as you type | | CSV export | ✅ Available | | | CSV upload | ⚠️ Limited | Upload CSV is available for record metadata spreadsheets, but not file. | | Autosave during grid session | ✅ Available | Drafts saved automatically | | Sync Changes to commit to Synapse | ✅ Available | Explicit submission step | | Permission access control | ✅ Available | Follows Synapse project permissions | | Curie (AI Grid Assistant) | ✅ Available | Chat assistance for cleaning, validating, and populating data | | Schema inspection (View Validation Schema) | ✅ Available | JSON schema view for advanced users | | Consortium-managed metadata schemas | ✅ Available | Applied by Sage Bionetworks for Data Coordination projects | | Browser search (Ctrl/Cmd + F) | ✅ Available | | | Simultaneous (multi-user) editing | ✅ Available | Available for grid sessions using one of the collaboration modes: **Share with assignees only** or **Share with all editors** | | Real-time user presence indicators | ✅ Available | | | Version history / rollback in UI | 🚧 Under Construction / Planned | Previous grid versions not viewable | | Shareable grid session URLs | ✅ Available | Available for grid sessions using one of the collaboration modes: **Share with assignees only** or **Share with all editors** | | Built-in grid search/filter tools | 🚧 Under Construction / Planned | Planned improvement over browser search | | Self-serve schema selection for individuals | 🚧 Under Construction / Planned | Standard schemas planned for non-consortium users | ### **The value I need is not in the valid-value list. What should I do?** If the term you need does not appear in the valid-value list, go ahead and submit your data using the appropriate value. Then, notify your data manager or consortium contact that the term is missing from the metadata dictionary. They can review the request and update the list if the term is appropriate to add. ### **Is Synapse updated immediately as I edit the grid?** No. The grid functions as a **draft workspace** . Your edits are saved as you work, but metadata is **not submitted to Synapse** until you click **Sync Changes** or **Apply Changes.** ### **Why do I only see the Tasks and Actions tab on certain Synapse projects?** The **Tasks and Actions tab** is hidden if there are no curation tasks. If you believe you should see the **Tasks and Actions tab**, please contact the appropriate portal help desk. * [AD Knowledge Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12) * [ALS Portal Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [ARK Portal Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/11) * [Classic Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [ELITE Portal](https://sagebionetworks.jira.com/servicedesk/customer/portal/12/group/34) * [HTAN Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/1) * [MC2 Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/10) * [NF-OSI Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/2) * [NAMHub Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/89) If none of these apply, please submit questions to the [Synapse Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9) ## Glossary **Curator** Synapse feature for completing metadata in a spreadsheet-style grid. **File metadata** Metadata attached directly to Synapse files (stored as annotations). **Metadata template** Configuration that defines which metadata fields/columns appear in a sheet. **Record set** Table containing a collection of related records (e.g., participants or samples). **Record metadata** Metadata stored in tables to describe study entities (e.g., participants, samples, specimens). ### **Schema** Definition that specifies the structure, fields, data types, relationships, and validation rules for metadata. [JSON schema](https://json-schema.org/) is a popular format. **Spreadsheet-style grid** Row-and-column interface for viewing and editing metadata. **Synapse Annotations** Public key--value metadata fields associated with Synapse entities. **Sync Changes** Button that commits grid edits to Synapse as submitted metadata. **Task** Guided action item for completing required or recommended project/data steps. **Tasks and Actions tab** Synapse project tab where tasks and related metadata sheets are accessed. ## Additional Resources **May 2026 Curator Webinar -** [**PDF**](https://drive.google.com/file/d/1Qus9B8I-DnheVg8gmOK5E-1XIROl-k9L/view?usp=sharing) --- language: "en" --- # Managing Your Account This page describes how to create, access, and manage your Synapse account, including authentication, account recovery, and security best practices. *** ** * ** *** Anyone can browse public content on Synapse, but you need an account to download and/or add content. To create an account, you must be over the age of 18 and have an email address. Synapse will send an email verification message to complete registration. Some actions in Synapse require additional steps, such as certification or validation. See [Synapse user account types](https://docs.synapse.org/synapse-docs/synapse-user-account-types) to learn which account type you need. *** ** * ** *** ## Creating a Synapse Account You can create a Synapse account in one of the following ways: * **Sign up** at * **Accept an invitation** to join a Synapse team, using the email link provided to you All Synapse accounts require **two-factor authentication (2FA)** to complete registration and log in. *** ** * ** *** ## Two-Factor Authentication (2FA) Two-factor authentication (2FA) adds an extra layer of security by requiring a **time-based one-time password (TOTP)** in addition to your login password. After you verify your email address, you will need to set up 2FA: 1. Scan the displayed QR code using a TOTP authenticator 2. Enter the 6-digit code to confirm 3. Save your backup codes (see below) 2FA is **required**for all Synapse accounts. ### Supported 2FA Methods Synapse supports **TOTP-based authenticators**, such as Authy, Duo mobile, Google, or Microsoft authenticators. SMS-based authentication is not supported. You may use: * Mobile authenticator apps * Desktop applications * Browser-based authenticator extensions A smartphone is **not required**. #### Using Synapse Without a Phone Some users do not have access to a smartphone or prefer not to install authenticator apps on mobile devices. Synapse supports **desktop and browser-based TOTP authenticators** that allow full account access without a phone. The following authenticators work on Windows, macOS, and Linux: * [Authenticator](https://authenticator.cc/) (free and open-source) * **Note:** Authenticator stores codes locally on your browser profile. If your browser is uninstalled or your computer is lost, you will lose access unless you have backup codes. * [Bitwarden](https://bitwarden.com/) (subscription required) * [LastPass](https://www.lastpass.com/) (free) * [1Password](https://support.1password.com/one-time-passwords/) (subscription required) ### Backup Codes After 2FA is set up, Synapse generates **one-time backup codes** . Store them in a secure place **separate from your authenticator**. ### 2FA Recovery **If you lost your authenticator but still have backup codes** Use a **backup code** at login, then reconfigure your authenticator after you regain access. (Backup codes are intended for exactly this scenario.) **If you lost both your authenticator and your backup codes** 1. Log in to [https://accounts.synapse.org](https://accounts.synapse.org/) by entering your password or completing the OAuth flow 2. Select **"Lost access to your codes?"** 3. Follow the prompts to send a reset email to the address associated with your account *** ** * ** *** ## Managing your profile Visit your **user profile** (click the letter icon or photo in the bottom-left and select **View Profile** ). From there, select **Edit Profile** to: * Change your Synapse username, email, or password * Add/edit your first and last name * Add/edit additional information (affiliation, title, etc.) * Upload a profile picture * Add a brief biography ### Account settings Open **Account Settings** (letter icon or photo → **Account Settings**) to manage additional preferences and features, including: * Email Addresses * Change Password * Date/Time Format * Trust \& Credentials * Two-factor Authentication (2FA) * Personal Access Tokens (PATs) * OAuth Clients * Privacy Preferences **Password safety note:** Do not reuse passwords from other sites. Use a unique password and a secure password manager. ### Email addresses and notifications **Add additional email addresses** Your Synapse account can have **multiple email addresses**. Each time you add a new email, Synapse sends a confirmation link to verify ownership. **Add a Google email address to enable "Sign in with Google" (SSO)** Synapse supports Google Single Sign On (OAuth 2.0). If you're already signed in to Google in your browser, you can sign in without entering a Synapse password once your Google email is connected. To enable this, create your Synapse account using your Google email, or add your Google email as a **secondary email** in **Account Settings → Email Addresses** After adding the Google email: 1. Sign out of Synapse 2. Sign back in using **Sign in with Google** on the Synapse login page ### Synapse email alias and notifications Synapse creates an email alias for you: `@synapse.org`. Synapse uses this alias as a relay to send/receive messages while keeping your registered email private. Rules to know: * To email `@synapse.org`, you must send from an email address registered on your Synapse account; otherwise, the message will bounce. * Synapse forwards your message to the recipient's registered email and replaces your address with your Synapse alias. * Synapse sends platform notifications (e.g., @mentions) to the single email you have set as your **primary email** (manage in Account Settings). *** ** * ** *** ## Logging in programmatically ### Personal Access Tokens (PATs) You can log in to the Synapse [command line](https://python-docs.synapse.org/), [Python](https://python-docs.synapse.org/), or [R clients](https://r-docs.synapse.org/articles/manageSynapseCredentials.html) using a **personal access token** instead of a username and password. Tokens are recommended because they can be revoked and scoped. **Protect your tokens -** Never hardcode tokensinto code. Use environment variables or secure storage. To create/manage PATs: 1. Go to **Account Settings** 2. Scroll to **Personal Access Tokens** 3. Click **Manage Personal Access Tokens** to view existing tokens or select **Create New Token** Where entering a 2FA code isn't possible (e.g., automated jobs), PATs are the recommended approach because they do not prompt for interactive 2FA entry. *** ** * ** *** ## Deactivated Accounts In order to meet compliance requirements, accounts are automatically deactivated after a 370-day period of latency. Accounts can also be deactivated by the Sage Admin team if suspected misuse, violation of Terms of Service, or Synapse Pledge is detected. ### Reactivating Accounts Deactivated accounts can only be reactivated by Sage. File a ticket with our [Service Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/16/create/863) if you need assistance with your account. ## Deleting Your Account ![:warning:](https://docs.synapse.org/__attachments/a_93107b38dee20346589f619b4ab80518f98457d70c73b1f9f85d0a5632257e3b/atlassian-warning?cb=14432459925d605e05cae2605cdfe666) **Account Deletion is Permanent** Sage Bionetworks is committed to honoring your privacy rights under applicable regulations, including the GDPR Right to Erasure (Right to Be Forgotten). If you wish to stop using Synapse, you can request account deletion by contacting us through our [virtual help desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/9). Upon deletion, your account will be disabled and you will no longer be able to log in. Your personal identifying information --- such as your name, email address, and profile details --- will be removed from your account record and replaced with anonymized data. Please be aware that all of your *public activities* will remain publicly viewable, and all of your activities within private projects remain viewable to the people within that project [Synapse](https://help.synapse.org/docs/Managing-Your-Account.2055405596.html), as these activity records are necessary to preserve the scientific integrity and audit trail of the research data hosted on the platform. For GDPR-related inquiries, including Data Subject Access Requests, you may also contact Sage Bionetworks' Privacy Officer at [privacyofficer@sagebionetworks.org](mailto:privacyofficer@sagebionetworks.org).. --- language: "en" --- # Navigating Synapse If you're new to Synapse, you may be looking for an overview of the site structure and components. Your main point of entry into Synapse activity will likely be your dashboard. This is where all of "your projects" will live, including projects that you have created, projects you have favorited, and projects that have been shared with you. Your dashboard is also the page from which you can create a new project. ![Screen Shot 2022-07-20 at 3.17.28 PM.jpg](https://docs.synapse.org/__attachments/a_543bd2020c1c026afd38a4cedaf7108895dfc5c3a527f0bcdf1ae5f91aa75adc/Screen%20Shot%202022-07-20%20at%203.17.28%20PM.jpg?cb=9b4db34928cac42136262c9911bee6e6) ## Synapse Toolbar You can navigate through the rest of Synapse using the toolbar on the left side. ![navigating synapse - toolbar.png](https://docs.synapse.org/__attachments/a_d14e64bfa8ef9023c91eae921a507352f19c790c4685f469c508e1226b693306/navigating%20synapse%20-%20toolbar.png?cb=96540da16d99b4c0dc98d79bd1993a44) ### Synapse Homepage Click this icon at any time to re-orient yourself in Synapse by returning to the homepage. ### Projects This tab mimics the information found in your dashboard - it simply provides a quick way of getting there. ### Favorites Throughout Synapse, you can favorite pretty much any item (project, file, folder, table, dataset, etc.) by clicking the star icon next to its name. This will add that item to your favorites list. ![favorites.png](https://docs.synapse.org/__attachments/a_4ba288eed826e6fd630432075a682afff6ea8ff1bda13445ad97de85bcb44896/favorites.png?cb=702b9d4cf4dfa7c0283bffb7c88ebe5a) ### Teams For information on teams, see [Teams](https://docs.synapse.org/synapse-docs/teams.md). ### Challenges For information on challenges and how to create and/or participate in one, see [Challenges](https://docs.synapse.org/synapse-docs/challenges.md). ### Download Cart As you add items to download from within Synapse or a portal, the number of items you have in your download cart will be indicated here. Click on this icon at any time to access your download cart and download the items to your computer. For more information on downloading data, see [Finding and Downloading Data](https://docs.synapse.org/synapse-docs/finding-and-downloading-data.md). ### Search Having trouble finding a specific project or file? Use the search bar. ### User Profile This is where you can view your profile and/or change your account settings. For information on managing your profile and account settings, see [Managing Your Account](https://docs.synapse.org/synapse-docs/managing-your-account.md). ### Help We have a variety of tools to help you out. Visit the [Help section](https://docs.synapse.org/synapse-docs/help.md) of this site to help guide you to the resource that will best serve your needs. --- language: "en" --- # Organizing Data With Tables Synapse tables are used to organize web-accessible, sharable, and queryable data. Tables may be queried and edited with the Synapse UI, as well as with the Synapse programmatic clients. This article guides you through the process of creating a table in Synapse. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For information about how to query a table, see [Querying Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/querying-tables-views-and-datasets.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more about creating tables and creating queries using one of the Synapse programmatic clients, see: * Tables in [Python Docs](https://python-docs.synapse.org/reference/tables/) * Tables in [R Docs](https://r-docs.synapse.org/articles/tables.html) ## Creating a New Table From a File Tables require structured data contained in a `.csv` or `.tsv` file. If your structured data is saved in an Excel format such as `.xls` or `.xlsx`, you must "save-as" to convert the file to a `.csv` before you proceed in Synapse. Navigate to the **Tables** tab in your project and select **Upload a Table** . If you provide a `.csv` or `.tsv` file, Synapse will infer your table schema based on the column headers. You may further customize the schema by selecting **Schema Options**. For very large files, it may take time for the table to be built and indexed completely before it can be viewed. You may navigate away from the table once you have clicked **Create**, and you will not lose any data. ## Creating an Empty Table You have the option to create an empty table by clicking on the **Tables Tools** menu and **Add Table** . To build a new table, you must specify the table structure, column by column. Select **Add Column** to specify each column's properties. |----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Column Name** | Choose a name for the column that will appear in the header. Column names must be 256 characters or less. There are three reserved words that cannot be used for Column Name: ROW_ID, ROW_VERSION, ROW_ETAG (case insensitive). | | **Column Type** | Select the type of data that will be entered into this column. For detailed descriptions of each column type, see the [REST docs](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/model/table/ColumnType.html). | | **Size** | For certain column types, such as String or Link, you must specify the maximum size of a single value. The default value is 50 characters, but you can limit the maximum size to be between 1 and 1000 characters. For column types that are lists (such as StringList or IntegerList), this number specifies the character maximum for all values in the column. | | **Max List Length** | For some list column types (such as StringList or IntegerList), you must specify a maximum list length. This number describes the maximum number of values that can appear in your list. For example, a Max List Length of 3 means that you may enter a list of up to three items. | | **Default Value** *(optional)* | Choose a default value to pre-populate in every new row of the column. Leave this field blank if you do not want to specify a default. | | **Restrict Values** *(optional)* | If you want to restrict the values for a particular column, enter the list of allowed values to create a dropdown menu. You can then select entries from this menu when adding table rows. Leave this field blank if you do not want to restrict the values. | | **Facet** | Select columns to be included in a faceted search to the left of your table. Choose **Values** to filter from a list of all possible entries for this column. Choose **Range** to filter with a slider, which is recommended for numeric values. Select a blank field to remove this column from the faceted search. | After you create an empty table, select **Table Tools** and**Upload Data to Table** to upload data from a `.csv` or `.tsv` file. The first line of your file must match the table structure specified. Alternatively, you may add rows and table data manually. To add, delete, or modify existing rows, click on **Bulk Edit Table Cell Values** (pencil icon) to edit rows. ![image-20230308-190840.png](https://docs.synapse.org/__attachments/a_767b884d40ced8a62c97d707861013f7e027b143cca72ca2bb10f917435a865c/image-20230308-190840.png?cb=b562c2dcb826a2c6edb2a0a15b22fe4f) Click the **+** sign to add rows. To delete rows, check the boxes of the rows and click the **Trash Can** icon. ## Modifying a Table Schema Select **Table Tools** , **Show Table Schema,** and then select the **Edit Schema** button to modify the existing table structure. From **Edit Schema**, you can delete columns, add new columns, and modify existing columns. ## Searching a Table You can search for data within a table in two ways. The default search is a simple search menu to the left of your table. Use the facets to filter your dataset and narrow down your search. Table data can also be retrieved by using a SQL-like query language either through the web portal or through the analytical clients. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) See [Querying Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/querying-tables-views-and-datasets.md) for more information. ## Deleting a Table To delete the entire table, click on the **Table Tools** menu and then select **Delete Table**. If you do not see this option, you do not have permission to delete the table. Contact an administrator for the project to get permission. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about permissions at [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). ## Adding Files to a Table In addition to structured data, you can also add individual files to a table in Synapse. In the example below, this feature is used to add image files to a table containing histology data. First, navigate to a table and add a new column for files. To add columns, click **Table Tools** , **Show Table Schema,** and then **Edit Schema** . From the pop-up window, click the **Add Column** button and set the **Column Type** as **File** . Click **Save** to exit from the Edit Columns window. Next, click the **Edit Query Result** s button (the pencil icon). In the column you just created, click the **upload icon** to add a file from your local computer. ![upload-table-file.png](https://docs.synapse.org/__attachments/a_ca89955c51b4ac518c133e60e6a1ae04e418ad5edb29fbf1a75b168bb8a34969/upload-table-file.png?cb=4519d73f0920b327cc5d5a89da5860a6) ## Versioning a Table You can create a version history for any table in Synapse. Versioning helps you keep a record of what changes you made to the table and when you made them. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on versioning a table, see [Versioning Tables, Views, and Datasets](https://docs.synapse.org/synapse-docs/versioning-tables-views-and-datasets.md). --- language: "en" --- # Participating in a Challenge This tutorial will teach you the steps for participating in a challenge. ## Challenge Registration If you do not have a Synapse account, see [Getting Started](https://docs.synapse.org/synapse-docs/getting-started.md) to learn more about Synapse and how to become a certified Synapse user. You **must** be registered for the challenge to submit and participate. The registration button can be found on the home page or**How to Participate** page for every challenge. In order to be fully registered for any challenge, you must have a Synapse account. In addition, DREAM Challenges require that you: (1) Become a [certified user](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md) (2) Agree to the DREAM Rules of participation (3) Agree to the Terms of Use to work with the challenge data ## Join or Create a Team We encourage you to form a team with other participants for the challenge. You can either join a team or create your own team of collaborators. See instructions on how to form a team [here](https://docs.synapse.org/synapse-docs/teams.md). Note that you **cannot** be on more than one team per challenge. Once you have submitted as a team or individual, you will not be able to submit as another team. If you decide to be part of a team, please register your team to the challenge; there will be a place to do this in every challenge wiki. ## Accessing Challenge Data The data stored on the challenge Synapse site can be accessed using the Synapse website or programmatically using the Synapse R or Python clients. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn all about using Synapse by browsing through this documentation site. You can start [here](https://docs.synapse.org/synapse-docs/getting-started.md). File descriptions are provided on the **Data Description** page in each challenge wiki. ## Submit to the Challenge You can submit to a challenge queue by using the R, Python, or via the web. All submissions must be first uploaded to Synapse. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For instructions on how to upload to a project, see [Uploading and Organizing Data Into Projects, Files, and Folders](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data-into-projects-files-and-folders.md) and/or [Setting Up a Project](https://docs.synapse.org/synapse-docs/setting-up-a-project.md). Most challenge queues will be labeled by `challengename-subchallenge#` as a challenge may have different questions that it may want you to answer. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about submitting to [evaluation queues](https://docs.synapse.org/synapse-docs/evaluation-queues.md). ## Share Ideas and Ask Questions Every challenge has a discussion forum for participants. See the **Discussion** tab on the challenge project page. The forum is a space for participants to ask any questions and raise ideas. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to use the discussion forum [here](https://docs.synapse.org/synapse-docs/discussion-forums.md). --- language: "en" --- # Provenance Reproducible research is a fundamental responsibility of scientists, but the best practices for achieving it are not established in computational biology. The Synapse provenance system is one of many solutions you can use to make your work reproducible by you and others. ## Overview of Synapse Provenance Provenance is a concept describing the origin of something. In Synapse, it is used to describe the connections between the workflow steps used to create a particular file or set of results. Data analysis often involves multiple steps to go from a raw data file to a finished analysis. Synapse's provenance tools allow users to keep track of each step involved in an analysis and share those steps with other users. The model Synapse uses for provenance is based on the [W3C provenance spec](https://www.w3.org/TR/prov-n/) where items are derived from an activity which has components that were used and components that were executed. Think of the used items as input files and executed items as software or code. Both used and executed items can reside in Synapse or in URLs such as a link to a GitHub commit or a link to a specific version of a software tool. You can use the command line, Python, and R to create and edit provenance relationships. In the web client, you can add provenance relationships once the file has been uploaded. Below is a Synapse visualization of provenance relationships that was created with the example in this guide using our programmatic and web clients. In this example, we have two scripts, one that generates random numbers and another that takes a list of numbers and computes their squares. The project's workflow resembles the provenance relationships. ![provenanceWorkflowDemo.png](https://docs.synapse.org/__attachments/a_374136937ab2babd34052bd4f741382766fe3ebba0feb97bda7589c4b559f7d1/provenanceWorkflowDemo.png?cb=985530920fb1a6aa82b78bc88eb4e311) ## Setting Provenance when Uploading a File Let's begin with a script that generates a list of normally distributed random numbers and saves the output to a file. For example, you have an R script file called [generate_random_data.R](https://www.synapse.org/#!Synapse:syn7205215) and you've saved the output to a data file called [random_numbers.txt](https://www.synapse.org/#!Synapse:syn7208917). We'll begin by uploading the files to Synapse and then set their provenance. ## Upload a File and Add Provenance For this example, we'll use a project that already exists ([*Wondrous Research Example* : syn1901847](https://www.synapse.org/#!Synapse:syn1901847/files/)). The [code file](https://www.synapse.org/#!Synapse:syn7205215) is saved in Synapse with synID `syn7205215`, so we'll upload the data file to this project, or in Synapse terminology, the project will be the parent of the new entities. As the [random_numbers.txt file](https://www.synapse.org/#!Synapse:syn7208917) was generated from the above script, we are going to specify this using provenance. There are a couple ways to set provenance information for a Synapse entity. The `used` and `executed` arguments specify resources used and code executed in the process of creating the entity. Code can be stored in Synapse (as we did in the previous step) or, better yet, linked by URL to a source code versioning system like GitHub or SVN. As an example, we'll specify two somewhat contrived sources of provenance: 1. Synapse entity by synID: [syn7205215](https://www.synapse.org/#!Synapse:syn7205215) (the code file) 2. URL to a page describing [normal distributions](http://mathworld.wolfram.com/NormalDistribution.html) **Web** The web client does not support setting provenance when uploading a file. Instead, upload the file first, then navigate to the file in your project. Click on the**File Tools** dropdown in the upper right hand corner and select **Edit File Provenance** . In the resulting pop-up, enter the relevant information. If you are entering an external URL as a reference, include the full URL path. In this example, you would enter `http://mathworld.wolfram.com/NormalDistribution.html`. **Command Line** synapse add random_numbers.txt --parentId syn1901847 --executed syn7205215 --used http://mathworld.wolfram.com/NormalDistribution.html Alternatively in the command line client, if you have downloaded the file, you can specify a local path as such: synapse add random_numbers.txt --parentId syn1901847 --executed ./generate_random_data.R --used http://mathworld.wolfram.com/NormalDistribution.html **Python** Python # Set provenance for data file generated by the script file data_file = File(path="random_numbers.txt", parent="syn1901847") data_file = syn.store(data_file, executed="syn7205215", used="http://mathworld.wolfram.com/NormalDistribution.html") **R** R # Set provenance for data file generated by the script file data_file <- File(path="random_numbers.txt", parent="syn1901847") data_file <- synStore(data_file, executed="syn7205215", used="http://mathworld.wolfram.com/NormalDistribution.html") Once the data file is uploaded, Synapse will provide the synID assigned to that file. In this case, the synID is `syn7208917`. ## Editing Provenance To continue our example above, we'll now add some new results from our initial data file. We're going to take the results in `random_numbers.txt` and square them. The script to square the numbers will be [square.R](https://www.synapse.org/#!Synapse:syn7209078), and we'll save the output to a data file, [squares.txt](https://www.synapse.org/#!Synapse:syn7209166). As with the previous example, the code file is already saved in Synapse, so we'll upload the data file and set its provenance. **Web** To update the provenance on a file, navigate to the **File** that you would like to update. Click on the**File Tools** dropdown in the upper right hand corner and select **Edit File Provenance**. In the resulting pop-up, enter the relevant information. **Command Line** # Add the data file to Synapse synapse add squares.txt -parentId syn1901847 # Set the provenance for newly created entity syn7209166 using synId synapse set-provenance -id syn7209166 -executed syn7209078 -used syn7208917 # Set the provenance for newly created entity syn7209166 using local path synapse set-provenance -id syn7209166 -executed ./square.R -used ./random_numbers.txt **Python** Python # Add the data file to Synapse squared_file = File(path="squares.txt", parentId="syn1901847") squared_file = syn.store(squared_file) # Set provenance for newly created entity syn7209166 squared_file = syn.setProvenance(squared_file, activity = Activity(used = "syn7208917", executed = "syn7209078")) # Provenance can also be set using local variables instead of looking up synIds squared_file = syn.setProvenance(squared_file, activity = Activity(used = data_file, executed = "syn7209078")) **R** R # Add the data file to Synapse squared_file <- File(path="squares.txt", parentId="syn1901847") squared_file <- synStore(squared_file) # Set provenance for newly created entity syn7209166 act <- Activity(name = "Squared numbers", used = "syn7208917", executed = "syn7209078") synStore(squared_file, activity=act) # Provenance can also be set using local variables instead of looking up synIds act <- Activity(name = "Squared numbers", used = data_file, executed = "syn7209078") squared_file <- synStore(squared_file, activity=act) ## Deleting Provenance To delete a provenance relationship, you must be the person who created the entity. **Web** Navigate to the entity you would like to delete provenance from (e.g. a file or folder). In this example, we are deleting provenance from a file. Select **File Tools** , then**Edit File Provenance** . In the list of **Used** and **Executed** , click the minus symbol ( ![minus](https://docs.synapse.org/__attachments/a_d1c39e91a31e507c468ec2cfc1fa5637ffa3a8a6e66573be3ef77790270e6b79/atlassian-minus?cb=b04d1998b2b6fc12820166ffbfbaa231) ) next to the URL or synID to delete each activity and **Save** your changes. **Command Line** Currently, deleting provenance is not supported in the command line client. **Python** # Delete provenance on entity syn123 delete_provenance = syn.deleteProvenance('syn123') **R** R # Delete provenance on entity syn123 deleteProvenance = synDeleteProvenance('syn123') ## Viewing Provenance **Web** Navigate to a file to view its provenance. Clicking on the triple dots above an entity will expand it to show the file's full provenance. ![expandProvenance.png](https://docs.synapse.org/__attachments/a_d8748b1a5bc2e1cb3e8fcb086d1cafe9cefc981d5b0a82df18023067ac7f6e7e/expandProvenance.png?cb=a9b82cc9db736d9de31a6fccbf6581e1) **Command Line** synapse get-provenance -id syn7209166 **Python** Python provenance = syn.getProvenance("syn7209166") provenance **R** R provenance <- synGetProvenance("syn7209166") provenance ## Reusing Provenance for Multiple Files An**activity** is a Synapse object that helps to keep track of what objects were used in an analysis step, as well as what objects were generated. Thus, all relationships between Synapse objects and an activity are governed by dependencies. That is, an activity needs to know what it 'used', and outputs need to know what activity they were 'generatedBy'. A couple of points for clarity: * An activity can 'use' many things (i.e. many inputs to an analysis) * Many outputs can be 'generatedBy' the same activity If an activity isn't assigned to an entity and then stored, a separate graph will be created for each file that the activity generated. The following example is used to assign the same activity to multiple files resulting in one provenance graph: **Web** Unfortunately, the web interface does not support assigning the same activity to multiple files. This action must be completed using either the R or the Python client. **Command Line** The command line currently does not support assigning the same activity to multiple files. **Python** Python # Code used to generate the file will be syn123456 # Files used to generate the information expr_file = syn.get("syn246810", download=F) filter_file = syn.get("syn135791", download=F) # Activity to assign to multiple files act = Activity(name="filtering", used=[expr_file, filter_file], executed="syn123456") syn.store(final_file, activity=act) # Get the activity now associated with an entity act = syn.getProvenance(final_file) # Now you can set this activity to as many files as you want (file1, file2, etc are Synapse Files) file_list = [file_1, file_2, file_3] file_list = map(lambda x: syn.store(x, activity=act), file_list) **R** R # Code used to generate the file will be syn123456 # Files used to generate the information expr_file <- synGet("syn246810", download=F) filter_file <- synGet("syn135791", download=F) # Activity to assign to multiple files act <- Activity(name="filtering", used=list(expr_file, filter_file), executed="syn123456") finalFile <- synStore(finalFile, activity=act) # Get the activity now associated with an entity act <- synGetProvenance(finalFile) # Now you can set this activity to as many files as you want (file1, file2, etc are Synapse Files) finalList <- c(file1, file2, file3) finalList <- lapply(finalList, function(x) synStore(x, activity=act)) --- language: "en" --- # Querying Tables, Views, and Datasets Not familiar with these terms? Find more information on [Tables](https://docs.synapse.org/synapse-docs/organizing-data-with-tables.md), [Views](https://docs.synapse.org/synapse-docs/views.md), and [Datasets](https://docs.synapse.org/synapse-docs/datasets.md). You can query tables, views, and datasets to find the data you need quickly. Views are especially useful for finding data that may be spread across multiple folders or projects. You can take advantage of querying via the Synapse UI, or using one of the programmatic clients. ## Querying Tables, Views, and Datasets via the Synapse UI ### Using Simple Search On the web, simple search mode displays radio buttons or sliders to the left of a table, view, or dataset. These are facets that you can use to search the data. Each facet corresponds to a column in the table, view, or dataset If the simple search is not visible on a table, view, or dataset, then no facets were specified by the owner of that table, view, or dataset. In this case, use the advanced search described below to query for data. To use simple search facets, navigate to a table, view, or dataset. Select the features you are interested in to filter the results. Note that the slider for range in simple search is inclusive, meaning the smallest and largest numbers in the selected range will be included in your search results. ![facet.png](https://docs.synapse.org/__attachments/a_d9ef4da3662b24d7abc9407827639414078a35c2096bf4b8f65b5d758459b7bc/facet.png?cb=e81bb8bdd52431654630e27f640bfeec) #### Setting Up Simple Search If you are the owner of a table, view, or dataset, you can choose what facets appear in a simple search menu. To select the facets, click on the **Tools** menu and select **Show View Schema,** **Show Table Schema** , or **Show Dataset Schema** . Click on the**Edit Schema** button to view the Edit Columns window. You can choose what facets appear by selecting **Values** or **Range** from the dropdown menus under the **Facet** option. **Values** can be thought of as categories whereas **Range** is a date or number. Selecting a blank field will remove the facet from the search window. ![Screen Shot 2021-04-01 at 3.33.07 PM.png](https://docs.synapse.org/__attachments/a_340178973137470b567d1bd4640caa0c34dc2832479bdab47e9bfb20b030f53e/Screen%20Shot%202021-04-01%20at%203.33.07%20PM.png?cb=e398b7d820e2d0a9c056cfe545f791c2) **Note:** If you change the Column Type in the schema, you must set its facet selection again. ### Using Advanced Search Queries The data within a table, view, or dataset can also be retrieved by using a SQL-like query language from the Synapse web interface. To query a table, view, or dataset, select the funnel icon in the upper righthand menu to reveal the advanced search bar. ![select-advance-search.png](https://docs.synapse.org/__attachments/a_396acd0e6762416e642131eb27501c9f405617cfdc41a8e10d719364bd5da591/select-advance-search.png?cb=980ad37aa90a5935214f9197a206d23a) By default, the advanced search bar will be pre-populated with a query to capture all of the data contained in the table, view, or dataset. For example: SELECT * FROM syn3079449 You may specify columns explicitly after the `SELECT` statement to subset the data: SELECT age, gender FROM syn3079449 To keep all columns in the query and filter rows that meet a certain condition, incorporate a `WHERE` statement: SELECT * FROM syn3079449 WHERE age > 50 Add an `ORDER BY` statement and a column name to sort the results by that column. The `ASC` statement sorts the data returned in ascending order. SELECT * FROM syn3079449 WHERE age > 50 ORDER BY "treatmentArm" ASC `COUNT`, `SELECT AS`, and `GROUP_CONCAT` SQL statements are also supported. To count the number of rows: SELECT count(*) FROM syn3079449 Select and rename a subset of columns: SELECT age AS "Age at Diagnosis", gender AS "Gender" FROM syn3079449 Group rows by name and identify distinct differences: SELECT count(distinct(treatmentArm)) AS "Number of Treatments", gender FROM syn3079449 group by gender To list out the distinct treatment arms that were studied, by gender: SELECT GROUP_CONCAT(distinct(treatmentArm) SEPARATOR ', ') AS "Available Treatments", gender as "By Gender" FROM syn3079449 group by gender See the [REST API docs](http://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html) for a list of all queries that can be performed. ### Toggling Between Simple and Advanced Search You can toggle from the simple search to the advanced search without losing your query results. For example, if you selected treatment arm `A`, age of `23:64`, and gender as `female`, the query will be preserved in the advanced search bar. However, this feature is unidirectional because the advanced search allows for parameters that are not available with facets. Therefore, switching from advanced to simple search will result in resetting the search query. Synapse will warn you before your search is reset. ![Reset-search-query-warning.png](https://docs.synapse.org/__attachments/a_62eac1b74f84dfd0d9be21265f61279743c9621bbbcca6edccfde84f7b6808c8/Reset-search-query-warning.png?cb=c0d019ebcc0829c504b154bc1f7c9981) **Warning:** When toggling back to simple search, the query will be reset. ## Querying Tables, Views, and Datasets Programatically Tables, views, and datasets can also be queried directly from the programmatic clients, which accept all of the SQL-like language used above. For example, to query for the contents of syn12345678: **Command Line** synapse query 'SELECT * FROM syn12345678' **Python** query = syn.tableQuery('SELECT * FROM syn12345678') **R** query <- synTableQuery('SELECT * FROM syn12345678') The expressions are the conditions for limiting a search. Every entity has properties useful for searching: * All entities (projects, files, folders, tables/views, Docker containers): `id`, `name`, `createdOn`, `createdBy`, `modifiedOn`, `modifiedBy`, `etag`, `type`, `parentId`, `benefactorId`, `projectId` * Versionable entities (files, table, views, datasets): `currentVersion` * Files only: `dataFileHandleId` Files also have `contentMd5`, `contentSize`, and `contentType` as properties. These properties are not available in a view and are not searchable. SELECT * FROM syn12345678 WHERE "id" = 'syn00012' For a complete list of example queries, see: [SQL Query Examples](https://rest-docs.synapse.org/rest/org/sagebionetworks/repo/web/controller/TableExamples.html) ### Finding Files in a Specific Project To find files in a specific project, [create a file view](https://docs.synapse.org/synapse-docs/views.md) in the web client. For example, if you'd like to see all files in a project, navigate to your project and then select the **Tables** tab. From there, click **Tables Tools** and **Add File View** . Click **Add container** and **Enter Synapse ID** to create a tabular file view that contains every file in the project, which you can now query. Importantly, if you want to later query on annotations, you must select **Add All Annotations**. ### Listing Files in a Specific Folder If you are using a programmatic client, you can list the files in a specific folder. First, you need to know the synID of the folder (for example syn1524884, which has data from TCGA related to melanoma). All entities in this folder will have a parentID of syn1524884. The function to find all files in this folder is called "getChildren": **Python** foo = list(syn.getChildren(parent='syn1524884', includeTypes=['file'])) **R** foo <- as.list(synGetChildren(parent='syn1524884', includeTypes=list('file'))) ### Queries on Annotations If annotations have been added to files, they can be used to discover files of interest from a file view syn12345678. For example, you can identify all files annotated as `bam` files (`fileFormat = bam`) with the following query: SELECT * FROM syn123456 WHERE "fileFormat"='bam' Likewise, if you put the RNA-Seq related files described in the section above into the project syn00123 with the described annotations, then you could find all of the files for `conditionB` and `sampleA`: SELECT * FROM syn123456 WHERE "projectId"='syn00123' AND "specimenID"='sampleA_conditionB' Lastly, you can query on a subset of entities that have a specific annotation. You can limit the annotations you want displayed as following. SELECT specimenID,genomeBuild,fileFormat,platform FROM file WHERE "projectId"='syn00123' AND "specimenID"='sampleA_conditionB' Reproducible queries can be constructed using one of the analytical clients (command line, Python, and R) and on the web client, query results can be displayed in a table on a wiki page. In a project, from the wiki page click **Wiki Tools** in the upper right corner to **Edit Project Wiki** . Click **Insert** and choose **Table: Query on Files/Folders** . Enter your query in the box and click the **Insert** button. Once you save the wiki page, the results will be displayed as a table. synapse query "SELECT specimenID,genomeBuild,fileFormat,platform FROM syn123456 WHERE \"specimenID\"='sampleA_conditionB'" result = syn.tableQuery("SELECT specimenID,genomeBuild,fileFormat,platform FROM syn123456 WHERE \"specimenID\"='sampleA_conditionB'") result = synTableQuery("SELECT specimenID,genomeBuild,fileFormat,platform FROM syn123456 WHERE \"specimenID\"='sampleA_conditionB'") ### Downloading from a Query You can download files in a folder using queries. Currently this feature is only available in the command line client. For example, if you want to download all files in a file view that has a synapse ID of `syn00123`, use: synapse get -q "SELECT * FROM file WHERE parentId = 'syn00123'" ## Troubleshooting Single quotes in Synapse queries must be replaced by double quotes or two single quotes. In order to query for the `chemicalStructure` of `4'-chemical`: SELECT * FROM syn123 where "chemicalStructure" = '4"-chemical' # OR SELECT * FROM syn123 where "chemicalStructure" = '4''-chemical' --- language: "en" --- # Release Notes Welcome to **Synapse Releases**, here is where we'll update you with releases and news related to the Synapse Platform and Portals, roadmap updates, and new feature notices. Use the panel on the left to browse each release. Synapse releases are usually published weekly on Sundays, with additional releases as needed for hotfixes. To determine which stack version is currently in production, scroll to the bottom (footer) of [synapse.org](http://synapse.org/) . The stack information is displayed there. ![image-20260225-155055.png](https://docs.synapse.org/__attachments/a_ea478ee5d85ad58c0a530b97006a4fb70b2c6aea70c88b432cf2545fb2135f3d/image-20260225-155055.png?cb=2a752af6280ae01afc59859c2a5ccd4f) --- language: "en" --- # Running a Challenge Synapse enables you to host challenges, providing an excellent avenue to crowd-source new computational methods for fundamental questions in systems biology and translational medicine. Setting up and running your own Challenge on Synapse is free\*! This tutorial will teach you the steps to hosting a Challenge on Synapse. For a visual walkthrough, you can also refer to [this Storylane](https://app.storylane.io/share/g0rzozz4q5ep). Need personalized support with data governance, infrastructure setup, or any other aspect of your Challenge? Our dedicated Challenges \& Benchmarking (CNB) team is here to help! Get started by outlining your Challenge plan [here](https://sagebionetworks.jira.com/servicedesk/customer/portal/18/group/27/create/202) and a CNB team member will be in touch. ~\*up to 100 GB data storage~ *** ** * ** *** ## Setting Up Your Challenge A Synapse Challenge always starts with two key Synapse entities: * **Participant Team:** A Synapse Team that serves as the central hub for challenge registration. Once users register for your Challenge, they will be automatically added to this team^+^. This makes it simple to communicate with all registered participants through the Synapse team email or by tagging the team name in a Discussion thread for announcements. ^+^the Team can be configured to require manager approval before participants are added * **Challenge Project:** A Synapse Project acts as your challenge's official "website". It is the primary place where participants can: * find all the crucial information, such as details about the challenge tasks and the scientific motivation behind them * access data (if hosted on Synapse) * make submissions * contact the challenge organizers and other participants For challenge data, you have flexibility: you can either host the data directly on Synapse, or if the data is hosted elsewhere, you can link to it from Synapse by creating a File Link. Keep reading to learn how to set up your Challenge, including how to create Teams, add Evaluation Queues and enable Challenge registration. ### Challenge Team(s) At a minimum, your Challenge requires a Synapse Team to be the "Registered Participants Team". You can either use an existing Synapse Team or create a new one specifically for this challenge. We recommend a clear, descriptive name like "*YOUR_CHALLENGE_NAME* Participants Team" for easy identification. If you have multiple organizers who are planning to actively manage the Challenge, we also suggest creating a separate "Organizers Team". Using this Team will greatly streamline internal communication, simplify permission updates, and make sharing resources among your organizing group more efficient. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to create and manage Teams [here](https://help.synapse.org/docs/Teams.1985446029.html). ### Challenge Project (aka the Website) Your Challenge "website" is technically a Synapse Project with the following features: * Evaluation Queues * Registration button ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To create a Synapse Project, see [Creating a Project](https://help.synapse.org/docs/Setting-Up-a-Project.2055471258.html#SettingUpaProject-CreatingaProject). #### Navigating your Challenge Project Every Synapse Project includes several tabs relevant to managing your Challenge: * **Wiki** - this is where you can provide details about your Challenge. * **Files** - (if hosted on Synapse) this is where you can share challenge data with participants. * **Tables** - this is where you can create tables to view and monitor submissions. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To add a leaderboard table into a Wiki page, see [Embed a Submission View in a Wiki Page](https://help.synapse.org/docs/Evaluation-Queues.1985151345.html#EvaluationQueues-EmbedaSubmissionViewinaWikiPage). * **Challenge** - (once enabled; see below) this is where you can create and manage queues for participants to submit their predictions or Docker images. * **Discussion** - this is where organizers and participants can interact and communicate with each other. #### Enable Evaluation Queues By default, Evaluation Queues (a Synapse feature for accepting submissions) are not enabled for Synapse Projects. To activate the Evaluation Queues feature: 1. Click on the **Project Tools** menu in the top-right corner of your Project, followed by **Run Challenge**. 2. A new window will appear, prompting you for a "Participant Team". Enter the name of a Synapse Team you've designated as the participants team for your Challenge. 3. Click **Create Challenge** to save. You'll then be directed to a new **Challenge** tab within your Project. Here, you can update the Registered Participants team as needed, and create or delete Evaluation Queues. The **Challenge** tab is only visible to Synapse users with "Admin" privileges to at least one Evaluation Queue. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more detailed information on how to create and manage the queues, see [Evaluation Queues](https://help.synapse.org/docs/Evaluation-Queues.1985151345.html). #### Add Challenge Registration ##### Participant Registration Challenge registration can be added to a Wiki page by leveraging the **Join Team Button** widget. To add challenge registration: 1. Navigate to the Wiki page where you'd like the registration button to appear. We recommend using the main Wiki page, as it's often the first page new users see. 2. Click the pencil icon to **Edit Project Wiki**. An editing window will open. 3. Click on **+Insert** then select **Join Team Button** : ![Adding a 'Join. Team Button' widget to a Wiki page.](https://docs.synapse.org/__attachments/a_a997b58c169c96692220d7836219bc7d4e49a8fb3b5825399bec9f3f73784bcb/join-team-widget.png?cb=c709f519cee200ea04dd73fb879da6da) 4. Another window will display. Complete the fields as prompted, ensuring to enter the **same** Synapse Team that is currently designated as the "Registered Participants Team" for your Challenge. 5. Before saving the widget configuration, make sure to enable the checkbox next to "Is this a challenge sign-up?" 6. Click **Save**, and a new Markdown will be provided in the editing window. It will look something like this: > `${jointeam?teamId=PARTICIPANT_TEAM_ID&isChallenge=true&isMemberMessage=Registered for CHALLENGE_NAME&text=Click Here to Register&isSimpleRequestButton=true&requestOpenText=Your registration is in progress%2E&successMessage=Your registration is in progress%2E}` 7. Click **Preview** to review the button placement; move around the Markdown as needed. 8. Click **Save** to finalize the changes onto the Wiki page. ##### Team Registration Once participants have registered individually, they can also register their team for your Challenge. For example: ![team-registration.png](https://docs.synapse.org/__attachments/a_e9ca107bbc58d8610b4766a08337fa75fae1f3cded32a5ee2627864a3f676fce/team-registration.png?cb=9ca391caebcb832d5d17fa3ac2343672) To add a team registration button to your Challenge: 1. Navigate to the Wiki page where you'd like the team registration button to appear. 2. Click on the pencil icon to **Edit Project Wiki**. An editing window will open. 3. Add the following Markdown, replacing *YOUR_CHALLENGE_ID* with the ID of your Challenge (which you can find in the **Challenges** tab) and *BUTTON_TITLE* with the text you want displayed on the button: > `${registerChallengeTeam?challengeId=YOUR_CHALLENGE_ID&buttonText=BUTTON_TITLE}` 4. Click **Preview** to review the button and its placement. Move around the Markdown as needed. 5. Click **Save** to finalize the changes onto the Wiki page. #### Upload Challenge Data For easier file management, we recommend creating Folders first and then uploading the data into them, rather than directly uploading Files into the project. For example: ![sample-files-structure.png](https://docs.synapse.org/__attachments/a_735c352ae0250576da881f16f064fce4552279e7e4a4f1e2fdecd9bbde4af8e7/sample-files-structure.png?cb=d1a2c7319cb5f899f0e5bfe5de467bdc) ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn how to create Folders and upload Files at [Uploading and Organizing Data](https://help.synapse.org/docs/Uploading-and-Organizing-Data-Into-Projects,-Files,-and-Folders.2048327716.html#UploadingandOrganizingDataIntoProjects,Files,andFolders-Files). Notice how these Folders (and subsequently, the Files) are marked as **Private**, as indicated by the lock icons. This is important because, generally, only specific users (like registered participants) should have access to the challenge data. Or even no access at all, as in the case of the "Groundtruth" Folder/Files. We'll cover how to set these permissions in more detail below! ### Example Challenge Projects * [DREAM Challenge Wiki Template 2.0](https://www.synapse.org/Synapse:syn18058986/wiki/588170) * [BraTS 2023 Challenge](https://www.synapse.org/Synapse:syn51156910) * [First DREAM Target 2035 Drug Discovery Challenge](https://www.synapse.org/Synapse:syn65660836) * and more *** ** * ** *** ## Launching Your Challenge To ensure your Challenge is discoverable, accessible, and open for registrations to anyone on the web, you will need to make your Project publicly viewable. By default, newly created Synapse Projects are only visible to the creator. To make your Challenge publicly viewable: 1. Click on the **Project Tools** menu, followed by **Project Sharing Settings**. 2. Click **Make public** , then adjust the access level for **All Synapse users**from "Can download" to "Can view". ‼️ **Important!** If your Project is hosting challenge data, please read the next section ++**before**++ making your Challenge public. ### **Important Considerations Before Launch** By default, all entities within your Synapse Project, like Files, Folders, and Submission Views, inherit the sharing settings of the main Project. However, for a challenge, you may only want ++registered++ participants to access and download the challenge data, not just anyone on the web. Therefore, before your Challenge goes live, you may need to update the permissions for certain Synapse entities. This includes, but isn't limited to: * Challenge data * Evaluation Queues By creating Local Sharing Settings for the data, you can ensure that the data won't become public, even after your main Challenge project does. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about [Local Sharing Settings](https://help.synapse.org/docs/Sharing-Settings,-Permissions,-and-Conditions-for-Use.2024276030.html#SharingSettings,Permissions,andConditionsforUse-EditSharingSettingsonFiles,Folders,andTables). #### Manage Challenge Data Permissions Whether your challenge data is organized into Folders or uploaded directly as Files, the steps are generally the same. Using Folders to organize your data is recommended, as setting the Local Sharing Settings on a Folder automatically applies those permissions to all Files within it, saving you the effort of individually setting permissions for each File. To update the permissions for each Folder or File containing challenge data: 1. Navigate to the **Files**tab of your Challenge project. 2. For each Folder or File containing challenge data: 1. Click on the Folder or File to navigate to its details page 2. Click on the **Folder Tools** or **File Tools** menu, then select **Folder Sharing Settings** (for a folder) or **File Sharing Settings** (for a file). 3. In the sharing settings window, click on **+ Create Local Sharing Settings**. 4. If necessary, revoke "Can download" access from all Synapse users and/or teams that should not have download permissions to the challenge data. *Note: "Can edit" and "Administrator" will also have download access, so adjust users with those roles as needed as well.* 5. Under **Add More**, enter the same Synapse Team currently designated as the "Registered Participants Team" for your Challenge, and set their permissions level to "Can download". 6. **Important:** If the File is a **ground truth file** , **do NOT share it with the Participants Team or with anyone** (unless they specifically need access, e.g. an organizer)**.** 7. Once you have made your changes, click **Save** to apply the new sharing settings. Here's an example of what sharing settings for a Folder containing challenge data might look like: ![sample-data-settings.png](https://docs.synapse.org/__attachments/a_82373582e107deb8745a5cb4cbf4a4eef914e9d30a6652593268f0f99838f3f0/sample-data-settings.png?cb=af6df472f428b5f42c71b7a2b7d1b73d) #### Manage Evaluation Queues Permissions Unlike other child entities, Evaluation Queues do not inherit settings from your main Project. By default, new queues are only accessible to the creator. To allow others to view results and make submissions, you will need to adjust their access settings: 1. Navigate to the **Challenge** tab of your Challenge project. 2. For each Evaluation Queue: 1. Click on the three dots, followed by **Modify Access**. 2. Click **Make public**. Ensure that "Anyone on the web" has "Can view" permissions. This will allow anyone on the web to see submissions and their results in a Submission View or leaderboard, without needing to sign into Synapse. 3. Under **Add More People**, enter: 1. the same **Synapse Team** designated as your Challenge's "Registered Participants Team" and set their permission level to "Can submit". 2. (if available) the Organizers Team for your Challenge, and set their permission level to "Can score". This allows the organizing group members to download the submissions. **Do NOT remove yourself as "Admin"** - doing so can cause irreversible changes that the Synapse IT team will be unable to assist with. Here's an example of what sharing settings for an Evaluation Queue might look like: ![sample-queue-settings.png](https://docs.synapse.org/__attachments/a_fd4c1960d810c529b274fa440599b6bde4e281121018126067e17a5c7ade6c2c/sample-queue-settings.png?cb=1ddad8f830bab9345c07cdd8edf90d73) ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn how to evaluate submissions, see [Evaluating Submissions](https://help.synapse.org/docs/Evaluating-Submissions.4156948561.html). *** ** * ** *** ## Closing Your Challenge After your Challenge has concluded, we recommend taking the following actions: 1. **Prevent new registrations.** To prevent users from joining the Participants Team and accessing challenge data after the challenge has ended: 1. From the Dashboard, click on the **Teams icon** from the left navigation bar. 2. Find or search for the Synapse Team designated as the "Registered Participants Team" for your Challenge. Click on the Team to go to its team page. 3. Click on the **Team Actions** menu, followed by **Edit Team**. 4. Under **Access**, select "Team is locked, users may not join or request access. New users must be invited by a team manager." 2. **Disable registration on your website.** Remove or hide all **Join Team Button** widgets to disable Challenge registration directly on the website. You can replace the registration button with an informative alert like this: ![challenge-closed-banner.png](https://docs.synapse.org/__attachments/a_6946353192c757ce31633dc0d93c9ff794f7a13f20640cfa617ee58f5ae3fe89/challenge-closed-banner.png?cb=21e41878a7dbbca1dd6a5a4970d88218) HTML
###! Challenge closed 🏆 Final results are [available here](YOUR_LINK_HERE). Thank you to all who participated!
To add this alert, paste the HTML above into the Wiki editing window. 3. **Stop active Evaluation Queue(s).** If any Evaluation Queues are still active, be sure to stop them. For safe measure, you can also remove "Can submit" permissions from the Participants team to prevent any post-challenge submissions. --- language: "en" --- # Sage Offerings **Note** : Are you preparing a data management plan as part of a grant proposal (for example for the **NIH** or **NSF** )? Sage can help you develop a **data management and sharing plan** that meets the requirements of the funder, develop a budget that meets the F.A.I.R principles, and provide guidance on how to store and secure your data. Find out more below and in the Frequently Asked Questions. ## Plans Sage offers the following service plans: 1. A free **Basic Hosting Plan** , intended for individual researchers wanting to share small datasets (\<100GB) for publication (including creating DOIs). This plan includes self-service project set-up and data hosting, and 25 hours of helpdesk support. [**Create an account on Synapse**](https://accounts.synapse.org/register1) for a free Basic Hosting plan. 2. A **Self-Managed Plan** , intended for researchers and labs looking for data longevity guarantees, consultation services for setting up and sharing data according to the F.A.I.R principles, access to consulting hours with our experienced governance team, 25 hours of helpdesk support, and tools for managing data access requests. [**Fill out a ticket**](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/16/create/162) to get a quote for a Self-Managed Plan. *This is our recommended plan for compliance with the NIH Data Management and Sharing Policy.* 3. A **Data Coordination Plan** , intended for multi-institutional research consortia looking for customized, end-to-end, collaborative data management. This plan supplements the services of the Self-Managed Planwith additional personalized consulting services, data curation, harmonization, and validation services. It also includes work to localize data policies and governance controls to your particular country's requirements, as applicable. A Data Coordination Plan includes a customized data exploration portal for your coordination center integrated with computational environments. If you are interesting in a Data Coordination Plan, [++**contact us**++](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/16/create/162) now for a quote. ### Features and pricing | | **Basic Hosting Plan** | **Self-Managed Plan** || |-----------------------------------|------------------------|------------------------|------------------------| | User Content Limit | **Up to 100GB** | **Up to 100GB** | **Up to 500GB** | | Storage Environment | **Shared Storage** | **Individual Storage** | **Individual Storage** | | Storage Retention | - | Supported for Duration of Plan || | Technical / Governance Consulting | - | 15 hours | 15 hours | | Data Management Plan Support | - | ✔️ | ✔️ | | Controlled Access | - | ✔️ | ✔️ | | Contract Length | - | 5 Years | 5 Years | | Plans Start At | **Free** | **USD $15,000** | **USD $30,000** | ### Features Shared by All Plans |-----------------------------------------------------------------|----| | DOIs for Publications \& Grants | ✔️ | | Access to Online Documentation \& Tutorials | ✔️ | | Project Wikis, Discussion Forum | ✔️ | | Support for Region-Specific Custom Storage Bucket in AWS or GCP | ✔️ | ## Frequently Asked Questions (FAQs) ### What is the 2023 NIH Data Management and Sharing Policy? The **National Institutes of Health (NIH)** [++**Data Management and Sharing Policy**++](https://sharing.nih.gov/data-management-and-sharing-policy/about-data-management-and-sharing-policies/data-management-and-sharing-policy-overview) (**DMSP**) went into effect on January 25, 2023. This policy applies to all research funded or conducted in whole or in part by NIH. The goal of the policy is to promote the sharing of scientific data, which can accelerate biomedical research discovery, enable validation of research results, and provide accessibility to high-value datasets. With a Managed Plan by Sage, we can help you with the following: * Develop a data management and sharing plan that meets the requirements of the NIH Data Management and Sharing Policy. * Develop an appropriate budget for your data management and sharing plan that meets the F.A.I.R principles. * Provide guidance on how to store and secure your data at a level appropriate for its sensitivity. * Promote the sharing of your data with the scientific community for scientific, education, and research purposes only. Note that other funders (beyond the NIH) now routinely require funded researchers to submit Data Management Plans. Sage can help you draft and execute your data management plan to comply with a number of funder requirements. ### What are the payment terms? **Basic Hosting Plan**: Access to the Service is provided free of charge. Sage reserves the right to implement fees for the Service, or portions thereof, at any time by providing you 30 days prior written notice on Synapse or otherwise. Before you become obligated to pay such fees, you will have the opportunity to export your data and terminate your account. The fees will only apply to users who have logged in or engaged in active use of the Service. **Self-Managed and Data Coordination Plans**: These plans and certain features of these plans require you to pay fees. All fees are in U.S. Dollars and are non-refundable, unless specified in the Sage Terms of Service or separate agreements. ### Do you offer more advanced and custom-tailored plans? For projects that require additional resources and services, we offer a **Data Coordination Plan**. The Data Coordination Plan has everything in our largest Self-Managed Plan, plus: * Unlimited data allowance and custom data storage locations * Fully managed project set-up, access and documentation * Tailor-made portal interface for data exploration * Seamless data integration with your existing computational tools and workflows * Customized governance support and specific geographical and institutional data policies * Dedicated Sage point of contact for unlimited, end-to-end support [++Contact us++](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/16/create/162) for more information about a custom Data Coordination Plan. ### How to get started? Sage's expertise makes it easy to create an **NIH Data Management and Sharing Plan** and budget. To support you, we provide budgeting information in the form of a quote, as well as sample text you can directly use in your plan. More questions? See our [FAQ](https://help.synapse.org/docs/FAQ.2047967233.html#FAQ-SageOfferings) ***Get started by*** [++***contacting us***++](https://sagebionetworks.jira.com/servicedesk/customer/portal/9/group/16/create/162)***now!*** Note: The Sage [Terms of Service](https://www.synapse.org/TrustCenter:TermsOfService) apply to each of these plans unless otherwise noted. Additional governance terms may apply. --- language: "en" --- # Setting Up a Project This tutorial will guide you through fundamental Synapse features for creating a project via the Synapse UI. You will learn how to: * Create your own project * Explore the project dashboard * Add a description to your project in the wiki * Share your work with other Synapse users, teams, or the public ## Prerequisites Anyone can browse public content on the Synapse website, but to create content using this tutorial, you will need to [register for an account](https://www.synapse.org/register) using your email address. You will receive an email message for verification to complete the registration process. To upload files to Synapse, you will need to perform the additional step of becoming a [certified user](https://docs.synapse.org/synapse-docs/synapse-user-account-types.md#Certified-Users). Because Synapse stores data from human subjects research, Sage Bionetworks requires that you demonstrate understanding of privacy and security issues. You can complete your certification by taking a short [certification quiz](https://www.synapse.org/#!Quiz:Certification) on Synapse. ## Creating a Project Projects are the main "containers" where information is stored and organized in Synapse. They are online workspaces where you can collaborate and share your work with teammates. Projects can be shared with individuals, small teams, or large consortia. Projects can be private so only you and your team can see what's inside, or they can be shared publicly for anyone to browse. To create a new project: 1. Click the **Projects** icon in the [Synapse toolbar](https://docs.synapse.org/synapse-docs/navigating-synapse.md) 2. Click the **plus sign (+)** next to **Projects** 3. Enter a unique name for your project and click **Save** Synapse will automatically open your new project so you can view your project dashboard. ## Exploring the Project Dashboard Your project dashboard allows you to see the different elements of your project at once. At the top of the project page, you can click the **star** next to your project name to add it to a list of favorites. ![Screen Shot 2022-07-20 at 6.47.27 PM.jpg](https://docs.synapse.org/__attachments/a_520cd1ac03b9544ac1745115757a02db0d4ce09e4fb396a485386b7f1c839e93/Screen%20Shot%202022-07-20%20at%206.47.27%20PM.jpg?cb=82ef4d5cd4f1d86405a4caf91d5efbe4) Below the project name, the **Synapse ID** (synID) is a unique number used to reference this project, and it is a powerful tool to identify and manage content. Projects, folders, files, tables, views, datasets, and wikis all have unique synIDs that can be used to navigate to and reference these specific items. The synID never changes and is always accessible in the URL and visible on the web. If you use one of the programmatic clients to interact with Synapse, you can use synIDs to create scripts that will work universally for anyone and without the need for specifying file paths. The project dashboard also has tabs for different sections of your project: * **Wiki**: A wiki is like a virtual notebook where you can describe your research so others can understand your goals, methods, or anything else you want to communicate about your project. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more, see [Wikis](https://docs.synapse.org/synapse-docs/creating-and-managing-wikis.md). * **Files:** This tab contains a directory of all of your files and folders within this project. Use this tab to see the hierarchy or structure of information as you add it. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more, see [Uploading and Organizing Data Into Projects, Files, and Folders](https://docs.synapse.org/synapse-docs/uploading-and-organizing-data-into-projects-files-and-folders.md). * **Datasets**: This tab contains any datasets that you have created. A dataset is a collection of files that already exist in Synapse that may be hosted in one or more Synapse projects or folders. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more, see [Datasets](https://docs.synapse.org/synapse-docs/datasets.md). * **Tables**: This tab contains tabular data. You can upload data tables or create them directly from the Synapse interface. You can also use this tab to create views, which are tables of other data in Synapse. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more about tables, see [Organizing Data With Tables](https://docs.synapse.org/synapse-docs/organizing-data-with-tables.md). ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more about views, see [Views](https://docs.synapse.org/synapse-docs/views.md). * **Discussion**: Use this tab to communicate with teammates in a discussion forum. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more, see [Discussion Forums](https://docs.synapse.org/synapse-docs/discussion-forums.md). * **Docker**: Docker is a platform for creating virtual containers to bundle code and other dependancies. You can add a Docker container to a project and share it with your teammates. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) To learn more, see [Synapse Docker Registry](https://docs.synapse.org/synapse-docs/synapse-docker-registry.md). ## Adding a Project Wiki A wiki is a virtual document that can be edited by multiple people on the web. Wikis are powerful tools to add information about your project, and Synapse offers over a dozen widgets to customize your wiki pages. Use wikis to provide descriptions of your project goals, methods, and data. Wiki pages can be written using text, Markdown, or basic HTML. Content can include images, tables, code blocks, LaTeX formatted equations, scholarly references, and references to other things in Synapse. To add wiki content: 1. Click on the **Wiki** tab and use the **Wiki Tools** menu. 2. Choose **Edit Wiki** to add or edit wiki content. 3. From the editing window, click**Preview** to check your work. Click **Save** when you are finished. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on how to organize and customize your wiki content, see [Creating and Managing Wikis](https://docs.synapse.org/synapse-docs/creating-and-managing-wikis.md). ## Sharing and Teams By default, your project and anything inside it are private, so only you can see it. However, you can share an entire project with specific users, teams, or make it public so anyone can browse your content. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) For more information on how Synapse can help you control access to your data, see [Sharing Settings, Permissions, and Conditions for Use](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use.md). To share a project: 1. Click on the **Project Settings** menu and select **Project Sharing Settings**. 2. Enter the usernames or team names to add collaborators. 3. To make the project publicly viewable, click **Make public** and then adjust the access levels for registered users and anyone on the web. Groups of users can form Synapse teams for collaboration. Teams can be used for managing permissions and for communicating with your collaborators. Sharing something with a single team instead of many individual users can help administrators manage large, complex projects. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about teams [here](https://docs.synapse.org/synapse-docs/teams.md) ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) And learn about using teams to manage group communication and project permissions at [Managing Data Access With Teams](https://docs.synapse.org/synapse-docs/managing-data-access-with-teams.md). Anything inside your project will automatically inherit the same sharing settings as the project itself. In other words, if your project is public, then all of the contents within the project will also be public. If needed, you can change these settings so that certain files or folders are only shared with specific groups of users. --- language: "en" --- # Sharing Settings, Permissions, and Conditions for Use As laid out in [Data Access Types](https://docs.synapse.org/synapse-docs/data-access-types), sharing settings and conditions for use are extra layers of protection that you can add to your data to determine who has access to it, and what they can do with it. ## Sharing Settings Use of sharing settings is included to all registered Synapse users as part of the Basic [++Synapse Plan++](https://docs.synapse.org/synapse-docs/sage-offerings). Sharing settings determine who can access content in Synapse and what permissions those users have with respect to a dataset. For example, sharing settings on a file can be used to control who can view, edit, download, or delete content. You are responsible for determining the appropriate sharing settings for any content that you upload into Synapse. ### Edit Sharing Settings on a Project When it comes to a project, there are two main options for sharing settings: public or private. The private sharing setting limits access to only specified users and teams. By default, all new Synapse projects are set to private, and you can manually add collaborators and/or allowed users as needed. You can then specify individual permissions for any of these private users. If you set your project to public, you can also specify permissions for any registered Synapse user, as well as to allow anyone on the web to view an item (more on permissions [below](https://docs.synapse.org/synapse-docs/sharing-settings-permissions-and-conditions-for-use#Edit-Sharing-Settings-on-a-Project)). To view and modify sharing settings on a project, go to that project and click **Project Settings** in the top right corner, and select **Project Sharing Settings**from the resulting dropdown menu. ![image-20230308-185249.png](https://docs.synapse.org/__attachments/a_e692b77fe858335afc5eae99d30954c8de37f25f9780dacca27af4f43b3bcc99/image-20230308-185249.png?cb=61dad5285f6ca3e7a50147ffc69e8571) In the **Project Sharing Settings**window, you can add individuals or groups by entering a username and then selecting the appropriate level of permissions from the dropdown menu. You can manage permissions for a group of users at once by first adding these users to a team. If certain individuals require unique permissions, consider creating multiple teams or sharing the item directly with the individuals that require unique permissions. ![plus](https://docs.synapse.org/__attachments/a_ee1c37efca34c61e1fc49d63a849e28b99f47172eab61cda828ee808d8227b5d/atlassian-plus?cb=252830f8c1dfea9d1e09927ba78d1476) Learn more about this: [Managing Data Access With Teams](https://docs.synapse.org/synapse-docs/managing-data-access-with-teams) At the bottom of the sharing settings pop-up window, you have the option to make the project public using the **Make Public** button. ![ProjectSharingSetting_2025.jpg](https://docs.synapse.org/__attachments/a_173bc40fcb1face9f249def6b3a491ca496b8183c8511c85f2087d20a98cea01/ProjectSharingSetting_2025.jpg?cb=35dcfc061c3b316794b39dc180af3dfa) Clicking on **Make Public** adds two additional groups to your sharing settings window: 1) all registered Synapse users and 2) anyone on the web. You can edit the level of access for either group and then click **Save** to make your changes. Note that you can only grant view permissions to "anyone on the web". To remove the settings for these two groups, click **Make Private**. ### Edit Sharing Settings on Files, Folders, and Tables You can adjust the sharing settings for individual folders, files, tables, and views separately from their parent project or folder. For example, you may wish to keep a particular folder private while you make the project public. Or you may want to share drafts of individual files with collaborators first prior to sharing them publicly. By default, all of the content residing within a parent project inherits the same sharing settings. If you move an item, then it carries those settings to another project. You can override this inheritance by defining a local sharing setting for that specific item. To do so, navigate to the file, folder, table, or view, then click on the**Tools** menu. Select the **Sharing Settings** option from the dropdown menu. You'll see the current (inherited) sharing settings in the resulting pop-up window. You will also see the option to **Create Local Sharing Settings**, which allows you to specify different sharing settings than the parent folder or project. ## Permissions In addition to applying global sharing settings on a project, file, folder, or table, you can also grant different levels of access, or permissions, to individuals or teams. The permission categories are: view, download, edit, edit and delete, and administrator. ### View Permissions View permissions give you the ability to see that something in Synapse exists (like the name of a project, file, folder, or table). You can discover the item using Synapse search, and it will be visible to you if included in a [table](https://docs.synapse.org/synapse-docs/organizing-data-with-tables) or a [file view](https://docs.synapse.org/synapse-docs/views#Creating-a-File-View). If there are [annotations](https://help.synapse.org/docs/Glossary.2667938103.html#Glossary-Annotations) associated with it, you can see these as well, however you cannot see the table or file contents. For example, if you have view permissions on a file, you will be able to see the file name and associated annotations, but you will not be able to see a preview of the file or download it. ### Download Permissions Download permissions give you the ability to see the contents of a project, file, folder, or table and download the contents to your own computer. Having download permissions includes also having view permissions. ### Edit Permissions Edit permissions allow a Synapse user to make changes to something in Synapse. This permission level also allows you to upload data to a folder or project where you are not an administrator (although you must be a [certified user](https://docs.synapse.org/synapse-docs/synapse-user-account-types#Certified-User) to do so). A user with edit permissions can: * Change the name of a project, folder, file, or table * [Change the annotations](https://docs.synapse.org/synapse-docs/annotating-data-with-metadata) associated with an entity, including removing existing annotations * [Create a new version of a file](https://docs.synapse.org/synapse-docs/versioning-files) * [Make changes to a wiki](https://docs.synapse.org/synapse-docs/creating-and-managing-wikis) * [Change the storage location settings](https://docs.synapse.org/synapse-docs/custom-storage-locations) of a project or folder Someone with edit permissions cannot delete something that is shared with them. Edit permissions are cumulative with view and download permissions. ### Edit and Delete Permissions Edit and delete permissions allow you to delete something that is shared with you, in addition to the edit permissions previously described. ### Administrator Permissions Administrator permissions allow you to change the sharing settings and metadata related to an entity. You can also change the friendly URL of a project. You can add, remove, or modify the sharing settings, including removing yourself. Administrator permissions are cumulative with edit and delete permissions. ## Conditions for Use Conditions for use are put in place to define/restrict *how* users who have permission to download data may use it. These conditions are placed on [++controlled access data++](https://help.synapse.org/docs/Data-Access-Types.2014904611.html#DataAccessTypes-ControlledAccessData). Conditions for use may include IRB approval or other restrictions that you define as the data contributor. Conditions for use are supported for projects with a Self-Managed or Data Coordination [++Synapse Plan++](https://help.synapse.org/docs/Sage-Offerings.2965078125.html). Conditions for use typically are structured to comply with the terms under which the data were collected or with other human subjects regulations. For example, human '-omic' data may have conditions for use imposed by informed consent requirements, legal contracts, or other privacy requirements. It is also appropriate to add conditions for use to data collected from "vulnerable" populations and to data that could potentially harm individuals or groups if misused. If you have any questions about whether conditions for use should be applied to your data, please contact our [Access and Compliance Team (ACT)](https://sagebionetworks.jira.com/servicedesk/customer/portal/8). You are responsible for determining if the data you would like to contribute is controlled data and therefore requires conditions for use. Carefully consider the specific risks and appropriate protections required before sharing your data on Synapse. If there are no ethical, legal or regulatory reasons to impose conditions for use, the data can be used for any lawful research purpose. We ask that data submitted to Synapse be de-identified or pseudonymized according to local law and regulations (for example, HIPAA or GDPR). Guidance on de-identification according to HIPAA rules can be found [here](http://www.hhs.gov/ocr/privacy). Conditions for use can be set at the project, folder, file and table level. We recommend grouping files that require the same conditions for use in a dedicated folder within your project. It is important to note that conditions for use cannot be set for Synapse wikis or discussion forums: they are not designed to house data and therefore do not have conditions for use as a feature. ### Examples of Conditions for Use Conditions for use vary broadly. Some examples include: * Specification of what type of research or analysis can be conducted on the data, for example, a data set may only be able to be used for breast cancer research * Specification of who can conduct research with the data set, for example, only researchers at non-profit institutions can use the data for research * Requirement that the data user submit an Intended Data Use statement, Data Use Certificate, or IRB Approval Letter prior to accessing the data. ### How to Add Conditions for Use To upload any data to Synapse, you must become a [certified user](https://docs.synapse.org/synapse-docs/synapse-user-account-types#Certified-User) first and have a [Managed Plan](https://docs.synapse.org/synapse-docs/sage-offerings). For data in a project included in a [Managed Plan](https://docs.synapse.org/synapse-docs/sage-offerings), once you complete the steps to becoming certified, you can upload your data and add conditions for use to limit how your data will be used by others. If you would like to set conditions for use for an entire project, please contact the [Synapse Access and Compliance Team (ACT)](https://sagebionetworks.jira.com/servicedesk/customer/portal/8) at [act@sagebase.org](mailto:act@sagebase.org) for assistance. By default, content within a folder or project inherits the conditions for use of the parent folder or project. As with sharing settings, you can set local conditions for use for individual folders, files and tables, but unlike sharing settings, you can only **add to** the existing parent project/folder's conditions for use. In other words, all content within a folder or project has, at a minimum, the conditions for use of its parent folder or project, and may have **additional** local conditions for use as needed. You cannot create a folder, file, or table that has fewer conditions for use than its parent or that has conditions for use that conflict with that of the parent. If you are a Self-Managed or Data Coordination Managed Plan customer, to set conditions for use for folders, files, and tables in Synapse, navigate to the item and click the **Add Conditions for Use** button. This will prompt you to send a message to the [++Access \& Compliance Team (ACT)++](https://sagebionetworks.jira.com/servicedesk/customer/portal/8). ![image-20230308-185425.png](https://docs.synapse.org/__attachments/a_d6d855d8bfcf9db02bf52b6553e648c11ae3e9ac88b3330302dd6a4a80c4e531/image-20230308-185425.png?cb=68c98a352d695e670c88c4df08a8b511) The resulting pop-up window will ask you if the data is sensitive human data that must be protected. If you answer **yes** , you will be prompted to contact the [++Access \& Compliance Team (ACT)++](https://sagebionetworks.jira.com/servicedesk/customer/portal/8) for assistance. If you answer**no** , but still feel that your data requires conditions for use, [++contact the ACT++](https://sagebionetworks.jira.com/servicedesk/customer/portal/8) to discuss your needs. Note that once you click the **Add Conditions for Use** button, your data will no longer be accessible to others until conditions for use have been established with the ACT. A "lock" will be placed on your data, which can only be removed by the ACT. ### How to Access Data with Conditions for Use To access data with conditions for use (controlled access data), you must be a [++registered user++](https://docs.synapse.org/synapse-docs/synapse-user-account-types#Registered-User), and you must fulfill the conditions for use set by the data contributor. This may include a requirement to be Certified and/or have a Validated profile. Navigate to the file, folder, or table, then look for the yellow key symbol. Click **Request Access** to open a dialog box with directions to meet the conditions for use. In many cases, you must read and electronically agree to data-specific terms. Occasionally, access to controlled data requires additional steps, like signing a Data Use Certificate (DUC), and/or having your protocol approved by an ethics board or IRB. ![image-20230308-185710.png](https://docs.synapse.org/__attachments/a_a56c97ef551fcdbac09e3648ad18051be24e3deb7954cd65eecd7f155dbc9652/image-20230308-185710.png?cb=eb5efdf9c0a4446031eb785379f893b7) Sharing a Synapse account or sharing controlled access data with other collaborators violates the Sage Bionetworks [Terms of Service](https://www.synapse.org/TrustCenter:TermsOfService). Each user wishing to access controlled data must individually agree to the Conditions for Use. Even if your collaborators have access to the same controlled data, be mindful when sharing information. Do not send data or metadata via email. **Warning:** Controlled access data may not be redistributed or shared. All users must individually agree to the conditions for use before accessing this type of data. Do not send controlled access data or metadata via email. ### Flagging Inappropriate Data Use If you believe data in Synapse is being shared inconsistently with the associated conditions for use, use the **Report Violation** flag in the header of the relevant file, folder, or table. ![image-20230308-185815.png](https://docs.synapse.org/__attachments/a_fb9de3e02223ff4c041669e159d6a79f674a8081683ac0d6962ec3d6a9feeb5c/image-20230308-185815.png?cb=6840d87c132358679c33fe34da9e3f30) Flagging the data will alert the Privacy, Security, \& Compliance Office (PSCO), and we will contact you for more information. You may also visit the [PSCO Help Center](https://sagebionetworks.jira.com/servicedesk/customer/portal/20) directly to report possible incidents. --- language: "en" --- # stack-560 Release Notes ## Summary stack-560 This release introduces significant enhancements to data management, curation, and user experience across Synapse, with a focus on improving the usability of the Data Grid, refining search functionality, and streamlining data access review workflows. We're delivering new features to empower Data Curators with better editing tools, providing ACT members with more efficient ways to manage data access submissions, and ensuring a more consistent and reliable platform experience for all users. These updates aim to increase efficiency for data curators and administrators, improve data integrity, and provide a smoother, more intuitive experience for users interacting with datasets and forum content. Number of tickets completed: 13 ## Affected Users \& Systems * **ACT Members**: Enhanced filtering for data access submissions. * **Data Curators**: Improved Data Grid functionality, including undo/redo capabilities and better CSV export behavior. * **All Synapse Users**: Improvements to discussion forum image rendering, search results consistency, and dataset versioning behavior. * **Developers/Integrators**: New API for Metadata Task CRUD operations, internal improvements to websocket communication. * **Synapse-React-Client**: Bundle size reduction for improved performance. ## Deprecated or Breaking Changes * **Grid CSV Export Behavior**: The CSV export of grid data will no longer convert "null" values to empty strings. Instead, "null" will now be exported as an empty field in the CSV, aligning with standard CSV representation of null/missing data. This change may affect automated parsing of exported CSVs that relied on "null" being an empty string. (PLFM-9224) ## New Features ### Data Grid Enhancements for Curators Users of the data grid will now have powerful new tools to manage their data: * **Undo/Redo Actions**: The Data Grid now supports undoing and redoing individual actions, providing greater flexibility and error correction capabilities for data curators. (SWC-7443, SWC-7451) * **Export from RecordSet Grid**: Users can now export a .csv file from a RecordSet Grid, which can then be used to update the RecordSet and create a new version. (PLFM-9197) * **Metadata Task CRUD Operations API**: A new API is available to support Create, Read, Update, and Delete operations for metadata tasks, facilitating more programmatic control over metadata curation workflows. (PLFM-9199) #### Improved Data Access Submission Management * **Filter for Unassigned Submissions**: ACT members can now filter data access submissions to easily view only those without an assigned reviewer, streamlining their workflow for claiming and managing reviews. This utilizes the POST /dataAccessSubmission/search.html endpoint. (SWC-7457) ## Fixes \& Improvements ### Discussion Forum \& Content Display * **Image Rendering in Discussion Forum**: Resolved an issue where images linked via Synapse IDs within the discussion forum's wiki widget were not rendering correctly. (SWC-7472) #### Data Grid Stability \& User Experience * **Pasting into Enum Cells**: Addressed bugs related to pasting values into enum cells within the data grid, improving reliability for data entry. (SWC-7453) * **Validation Styles for Unbound Grids**: The validation spinner effect will no longer be shown in the data grid when no schema is bound to the grid session, preventing unnecessary visual indicators. (SWC-7446) * **Large Patch Handling**: Implemented a mechanism to split very large data patches before sending them over the websocket connection, preventing errors due to data size limits during operations like pasting many rows. (SWC-7454) #### Search and Navigation * **OpenSearch Quote Handling**: Fixed an issue in OpenSearch where quoted and unquoted search terms produced identical results, ensuring that using quotes now narrows search results as expected. (PLFM-9218) * **Dataset Versioning Default**: Links to DataSets without a specified version will now default to the latest stable version, even for users with edit permissions, improving the default navigation experience. (SWC-6932) #### Performance * **Synapse-React-Client Bundle Size**: Reduced the bundle size of the synapse-react-client to improve overall performance, benefiting page load times and SEO. (SWC-6990) \> These release notes are auto generated by Google GEMINI so responses may not be completely accurate --- language: "en" --- # stack-561 Release Notes ## Summary stack-561 This release focuses on significant improvements to data management and user experience, particularly around how "Grid" data is handled, the introduction of "RecordSets" as a new entity type, and enhanced support for Docker repositories. We've also addressed several critical bugs impacting search functionality and system performance. These updates are designed to improve scalability, make data more discoverable and manageable, and provide a more robust and intuitive platform for all users. This work was prioritized to enhance platform stability, introduce new data organization capabilities, and refine user interaction with complex data structures. The expected impact includes faster access to grid data, improved search reliability, new ways to organize and present data through RecordSets, and a more streamlined process for managing Docker repositories. **10 tickets completed** ## Affected Users \& Systems * \***All Users:**\* Will benefit from improved search reliability and overall system performance. * \***Users working with Grid data:**\* Will experience faster data retrieval and improved scalability when interacting with large grids. * \***Data Curators/Managers:**\* Will gain new capabilities with "RecordSets" for organizing and presenting data, and enhanced flexibility in managing Docker repositories. * \***Access Requirement Teams:**\* Will have improved tools for communicating rejection reasons with clearer formatting. * \***Internal Systems:**\* The underlying database and Grid replica systems have been optimized for better performance and reliability. ## Deprecated or Breaking Changes **No breaking changes are introduced in this release.** ## New Features ### Introducing RecordSets for Enhanced Data Organization (SWC-7467) We're launching RecordSets, a new entity type designed to improve how you organize and present your data. RecordSets will have a user-friendly name, appear in the file hierarchy, and be discoverable via the EntityFinder. They will also feature a dedicated detail page, similar to a FileEntity page, allowing you to start new grid sessions directly from a RecordSet. Please note, it will not be possible to upload new versions of a RecordSet directly. #### New "Mentions" Metadata for Entities (SWC-7470) To enrich entity metadata, we've added a new "mentions" field. This feature will expose external references to entities, similar to how citations are displayed, providing richer context and discoverability for your data. #### UI for Grid Agent Session (SWC-7452) A new user interface is now available for initiating and managing Grid agent sessions. This feature will allow users to start an agent session using a gridSessionId within the draggable dialog. #### Enhanced Docker Repository Management (SWC-7413) When creating an external Docker repository, users now have the ability to create a commit directly. This streamlines the process of managing Docker repositories and ensures that new repositories are ready for use more quickly. Additionally, error messages related to missing commits have been improved for clarity. #### Update Grid Model with Current Selection (SWC-7471) The grid model has been updated to reflect the current selection, improving the responsiveness and accuracy of grid interactions. ## Fixes \& Improvements ### Improved Grid Internal Replica Scalability (PLFM-9243) Addressed a performance issue with the internal replica of grids, particularly affecting larger grids. We've optimized the database queries used to serve grid data by implementing generated columns and secondary indexes, significantly improving the scalability and speed of data retrieval for grid operations. #### Resolved Document Search Drop (PLFM-9249) Fixed an issue where document search results dramatically dropped around 9/16/2026. This resolves an intermittent but significant problem impacting the discoverability of documents for users. #### Resolved Timed Out Patch Acceptance in Grid Logs (PLFM-9178) Addressed an issue causing an excessive number of "Timed out waiting for a patch to be accepted" messages in the logs, indicating a performance bottleneck in the Grid replica patching process. This fix improves the reliability and efficiency of grid data synchronization. #### Line Breaks in Access Request Rejection Reasons (SWC-7466) Users can now add line breaks when editing access request rejection reasons (syn50681489). This enhancement allows for clearer and more readable communication of rejection details to applicants. \> These release notes are auto generated by Google GEMINI so responses may not be completely accurate [Next Page](https://docs.synapse.org/llms-full.txt/1)