Files and Versioning

Files

Synapse Files can be created by uploading or linking to digital files on the web. They are accessible to anyone who has access, can be annotated with custom metadata, can be embedded into Synapse Wiki pages, and can be associated with a DOI. Files carry the Conditions for Use of the Synapse Folder they are placed in, plus any additional specific Conditions for Use they have on their own.

By default, Files uploaded to Synapse are stored in ‘Synapse Storage’, which is freely available to you. Files can also be stored on your own Amazon S3 bucket (see Custom Storage Locations) or other custom locations. Furthermore, if you don’t want to upload a file (it has external restrictions on sharing, is really large, for example) you can also link to the file. In this way, the file will be accessible through the Synapse clients when you are on the computer that the file is stored, but can be annotated, queried, and documented with a Wiki through Synapse. Lastly, you can provide web-accessible links as Synapse files, which will redirect to that location. All of the same Synapse File features are available are available on external links as well.

Synapse Files (as well as Folders and Projects) are identified by a unique identifier called a Synapse ID. It takes the form syn12345678. This identifier can be used to refer to a specific file on the web and through the clients.

Uploading a File

# Add a local file to an existing project (syn12345) on Synapse
synapse store raw_data.txt --parentId syn123456

Moving a File

All Synapse clients offer a way to move files and folders. Please note that File Views and sync manifests cannot be used to move files.

The command line client has a sub-command mv which can be used to move files and folders. The Python and R clients do not have a specific move function, but can be used to modify the parentId property of the file/folder to move it. In the web client, there is an option in the Tools menu to move files or folders.

# move a file or folder (syn123) to a different folder/project (syn456)
synapse mv --id syn123 --parentId syn456

File Previews

Some files in Synapse are supported with previews to allow users to peek at the contents of the file before they download it. File Previews can also be embedded in Wikis. A list of what kinds of files are supported with rich previews will be added soon.

Versions of Files

Versioning is an important component to reusable, reproducible research. When a Synapse File is initially uploaded, it automatically gets a version of 1. It can be referred to explicitly by its Synapse ID: syn12345678.1. Uploading a new version of a file replaces the existing file in Synapse while preserving the previous version. The Synapse ID will remain but the version will increase, e.g., syn12345678.2. All versions are accessible through a single entry point (the Synapse ID, syn12345678). It is important to note that, by default, any previous versions of the file should still be available - they may be used in provenance relationships or as part of a data release.

Providing the Synapse ID without any versioning information to any of the clients (e.g., syn12345678) will always point to the most recent version of the file. In this way, updates to files can be automatically fetched by users by omitting the version.

If a DOI has been created for a Synapse file, it is automatically versioned as well, so specific versions can be cited in other places.

The easiest way to create a new version of an existing Synapse File is to use the same file name and store it in the same location (e.g., the same parentId). Synapse will automatically determine that a new version of a file is being stored, only if the contents of the file have changed. If the contents have not changed (e.g., the md5sum of the file is identical to the most recent version), a new file will not be uploaded and the version will not increase.

Only the file and annotations information are included in the version. Other metadata about a Synapse File (such as the description, name, parent, ACL, and its associated Wiki) are not part of the version, and will not change between versions.

Uploading a New Version

Uploading a new version follows the same steps as uploading a file for the first time - use the same file name and store it in the same location (e.g., the same parentId). It is recommended to add a comment to the new version in order to easily track differences at a glance. The example file raw_data.txt will now have a version of 2 and a comment describing the change.

# Upload a new version of raw_data.txt 
synapse store raw_data.txt --parentId syn123456 
#Currently there is no option to add a version comment when uploading via command line. We recommend adding the comment via the web client.

Updating Annotations or Provenance Without Changing Versions

Any change to a File will automatically update its version. If this isn’t the desired behavior, such as minor cahnges to the metadata, you can set forceVersion=False with the Python or R clients. For command line, the commands set-annotations and set-provenance will update the metadata without creating a new version. Adding/updating annotations and provenance in the web client will also not cause a version change.

Important: Because Provenance is tracked by version, set forceVersion=False for minor changes to avoid breaking Provenance.

Setting annotations without changing version

# Set annotation on file (syn56789)
synapse set-annotations --id syn56789 --annotations '{"fileType":"bam", "assay":"RNA-seq"}'


Setting provenance without changing version

# Setting provenance (syn56789)
synapse set-provenance -id syn56789 -executed ./path/to/example_code

Downloading a Specific Version

By default, the File downloaded will always be the most recent version. However, a specific version can be downloaded by passing the version parameter.

# Retrieve the first version of a file from Synapse
synapse get syn56789 -v 1


See Also

Provenance, Annotations and Queries, Downloading Data

Need More Help?

Try posting a question to our Forum.

Let us know what was unclear or what has not been covered. Reader feedback is key to making the documentation better, so please let us know or open an issue in our Github repository (Sage-Bionetworks/synapseDocs).