Overview

The synapseutils package provides both higher level functions as well as utilites for interacting with Synapse. These funtionalities include:

copy

synapseutils.copy.changeFileMetaData(syn, entity, downloadAs=None, contentType=None)
Parameters:
  • entity – Synapse entity Id or object
  • contentType – Specify content type to change the content type of a filehandle
  • downloadAs – Specify filename to change the filename of a filehandle
Returns:

Synapse Entity

Can be used to change the fileaname or the file content-type without downloading.

>>> e = syn.get(synid)
>>> print(os.path.basename(e.path))  ## prints, e.g., "my_file.txt"
>>> e = synapseutils.changeFileMetaData(syn, e, "my_newname_file.txt")
synapseutils.copy.copy(syn, entity, destinationId, skipCopyWikiPage=False, skipCopyAnnotations=False, **kwargs)
  • This function will assist users in copying entities (Tables, Links, Files, Folders, Projects), and will recursively copy everything in directories.
  • A Mapping of the old entities to the new entities will be created and all the wikis of each entity will also be copied over and links to synapse Ids will be updated.
Parameters:
  • syn – A synapse object: syn = synapseclient.login()- Must be logged into synapse
  • entity – A synapse entity ID
  • destinationId – Synapse ID of a folder/project that the copied entity is being copied to
  • skipCopyWikiPage – Skip copying the wiki pages Default is False
  • skipCopyAnnotations – Skips copying the annotations Default is False

Examples:: import synapseutils import synapseclient syn = synapseclient.login() synapseutils.copy(syn, ...)

Examples and extra parameters unique to each copy function – COPYING FILES

Parameters:
  • version – Can specify version of a file. Default to None
  • updateExisting – When the destination has an entity that has the same name, users can choose to update that entity. It must be the same entity type Default to False
  • setProvenance – Has three values to set the provenance of the copied entity: traceback: Sets to the source entity existing: Sets to source entity’s original provenance (if it exists) None: No provenance is set

Examples:: synapseutils.copy(syn, “syn12345”, “syn45678”, updateExisting=False, setProvenance = “traceback”,version=None)

– COPYING FOLDERS/PROJECTS

Parameters:excludeTypes – Accepts a list of entity types (file, table, link) which determines which entity types to not copy. Defaults to an empty list.

Examples:: #This will copy everything in the project into the destinationId except files and tables. synapseutils.copy(syn, “syn123450”,”syn345678”,excludeTypes=[“file”,”table”])

Returns:a mapping between the original and copied entity: {‘syn1234’:’syn33455’}
synapseutils.copy.copyFileHandles(syn, fileHandles, associateObjectTypes, associateObjectIds, contentTypes, fileNames)

Given a list of fileHandle Objects, copy the fileHandles

Parameters:
  • fileHandles – List of fileHandle Ids or Objects
  • associateObjectTypes – List of associated object types: FileEntity, TableEntity, WikiAttachment, UserProfileAttachment, MessageAttachment, TeamAttachment, SubmissionAttachment, VerificationSubmission (Must be the same length as fileHandles)
  • associateObjectIds – List of associated object Ids: If copying a file, the objectId is the synapse id, and if copying a wiki attachment, the object id is the wiki subpage id. (Must be the same length as fileHandles)
  • contentTypes – List of content types (Can change a filetype of a filehandle).
  • fileNames – List of filenames (Can change a filename of a filehandle).
Returns:

List of batch filehandle copy results, can include failureCodes: UNAUTHORIZED and NOT_FOUND

synapseutils.copy.copyWiki(syn, entity, destinationId, entitySubPageId=None, destinationSubPageId=None, updateLinks=True, updateSynIds=True, entityMap=None)

Copies wikis and updates internal links

Parameters:
  • syn – A synapse object: syn = synapseclient.login()- Must be logged into synapse
  • entity – A synapse ID of an entity whose wiki you want to copy
  • destinationId – Synapse ID of a folder/project that the wiki wants to be copied to
  • updateLinks – Update all the internal links. (e.g. syn1234/wiki/34345 becomes syn3345/wiki/49508) Defaults to True
  • updateSynIds – Update all the synapse ID’s referenced in the wikis. (e.g. syn1234 becomes syn2345) Defaults to True but needs an entityMap
  • entityMap – An entity map {‘oldSynId’,’newSynId’} to update the synapse IDs referenced in the wiki Defaults to None
  • entitySubPageId – Can specify subPageId and copy all of its subwikis Defaults to None, which copies the entire wiki subPageId can be found: https://www.synapse.org/#!Synapse:syn123/wiki/1234 In this case, 1234 is the subPageId.
  • destinationSubPageId – Can specify destination subPageId to copy wikis to Defaults to None
Returns:

A list of Objects with three fields: id, title and parentId.

walk

synapseutils.walk.walk(syn, synId)

Traverse through the hierarchy of files and folders stored under the synId. Has the same behavior as os.walk()

Parameters:
  • syn – A synapse object: syn = synapseclient.login()- Must be logged into synapse
  • synId – A synapse ID of a folder or project

Example:

walkedPath = walk(syn, "syn1234")

for dirpath, dirname, filename in walkedPath:
    print(dirpath)
    print(dirname) #All the folders in the directory path
    print(filename) #All the files in the directory path

sync

synapseutils.sync.generateManifest(syn, allFiles, filename)

Generates a manifest file based on a list of entities objects.

Parameters:
  • allFiles – A list of File Entities
  • filename – file where manifest will be written
synapseutils.sync.readManifestFile(syn, manifest_file)

Verifies a file manifest and returns a reordered dataframe ready for upload.

Parameters:
  • syn – A synapse object as obtained with syn = synapseclient.login()
  • manifest_file – A tsv file with file locations and metadata to be pushed to Synapse. See below for details

:returns a pandas dataframe if the manifest is validated.

See also for a description of the file format: - synapseutils.sync.syncToSynapse()

synapseutils.sync.syncFromSynapse(syn, entity, path=None, ifcollision=u'overwrite.local', allFiles=None, followLink=False)

Synchronizes all the files in a folder (including subfolders) from Synapse and adds a readme manifest with file metadata.

Parameters:
  • syn – A synapse object as obtained with syn = synapseclient.login()
  • entity – A Synapse ID, a Synapse Entity object of type folder or project.
  • path – An optional path where the file hierarchy will be reproduced. If not specified the files will by default be placed in the synapseCache.
  • ifcollision – Determines how to handle file collisions. May be “overwrite.local”, “keep.local”, or “keep.both”. Defaults to “overwrite.local”.
  • followLink – Determines whether the link returns the target Entity. Defaults to False
Returns:

list of entities (files, tables, links)

This function will crawl all subfolders of the project/folder specified by entity and download all files that have not already been downloaded. If there are newer files in Synapse (or a local file has been edited outside of the cache) since the last download then local the file will be replaced by the new file unless ifcollision is changed.

If the files are being downloaded to a specific location outside of the Synapse Cache We will also add a file (SYNAPSE_METEDATA_MANIFEST.tsv) in the path that contains the metadata (annotations, storage location and provenance of all downloaded files)

See also: - synapseutils.sync.syncToSynapse()

Example:: Download and print the paths of all downloaded files:

entities = syncFromSynapse(syn, "syn1234")
for f in entities:
    print(f.path)
synapseutils.sync.syncToSynapse(syn, manifest_file, dry_run=False, sendMessages=True, retries=4)

Synchronizes files specified in the manifest file to Synapse

Parameters:
  • syn – A synapse object as obtained with syn = synapseclient.login()
  • manifest_file – A tsv file with file locations and metadata to be pushed to Synapse. See below for details
  • dry_run – Performs validation without uploading if set to True (default is False)

Given a file describing all of the uploads uploads the content to Synapse and optionally notifies you via Synapse messagging (email) at specific intervals, on errors and on completion.

Manifest file format

The format of the manifest file is a tab delimited file with one row per file to upload and columns describing the file. The bare minimum required columns is path and parent where path is the local file path and parent is the synapse Id of the project or folder where the file is uploaded to. In addition to these columns you can specify any of the parameters to the File constructor (name, synapseStore, contentType) as well as parameters to the syn.store command (used, executed, activityName, activityDescription, forceVersion). Used and executed can be semi-colon(;) separated lists of synapse ids, urls and/or local filepaths of files already stored in Synapse (or being stored in Synapse by the manifest). Any additional columns will be added as annotations.

Required Fields:

Field Meaning Example
path local file path or URL /path/to/local/file.txt
parent synapse id syn1235

Common Fields:

Field Meaning Example
name name of file in Synapse Example_file
forceVersion whether to update version False

Provenance Fields:

Field Meaning Example
used List of items used to generate file syn1235; /path/to_local/file.txt
executed List of items exectued https://github.org/; /path/to_local/code.py
activityName Name of activity in provenance “Ran normalization”
activityDescription Text description on what was done “Ran algorithm xyx with parameters...”

Annotations:

Any columns that are not in the reserved names described above will be intepreted as annotations of the file

Other Optional fields:

Field Meaning Example
synapseStore Boolean describing wheterh to upload files True
contentType content type of file to overload defaults text/html

Example Manifest file

path parent annot1 annot2 used executed
/path/file1.txt syn1243 “bar” 3.1415 “syn124; /path/file2.txt” https://github.org/foo/bar
/path/file2.txt syn12433 “baz” 2.71 “” https://github.org/foo/baz

monitor

synapseutils.monitor.notifyMe(func, syn, messageSubject=u'', retries=0)

Function decorator that notifies you via email whenever an function completes running or there is a failure.

Parameters:
  • syn – A synapse object as obtained with syn = synapseclient.login()
  • messageSubject – A string with subject line for sent out messages.
  • retries – Number of retries to attempt on failure (default=0)
Example::

import synapseutils import synapseclient syn = synapseclient.login()

my_query = notifyMe(syn.tableQuery, syn, ‘Long running query’, retries=2) results = my_query(“select id from syn1223”)

synapseutils.monitor.with_progress_bar(func, totalCalls, prefix=u'', postfix=u'', isBytes=False)

Adds a progress bar to calls to a function

Parameters:
  • func – Function being wrapped with progress Bar
  • totalCalls – total number of items/bytes when completed
  • prefix – String printed before progress bar
  • prefix – String printed after progress bar
  • isBytes – A boolean indicating weather to convert bytes to kB, MB, GB etc.