File Transfer to Local Workstation

Rescale platform offers several ways to transfer files to your local workstation

After your job has finished, the Results pane displays all the files associated with the job.

Go to the Results pane for your job

results-pane-main.png

On the Results pane, the options to download files are:


Download entire job


Click on Download Job to download a zip file containing all files associated with the run, including input and output files

results-pane-download-job.png


Search for files


Type the file name or extension in the search bar. Click on Search. Click in the check box. Click on the Actions dropdown button and then click Download selected files

Note: If a single file is selected, the file itself will be downloaded. If more than one file is selected, the files will be compressed first and then download (the compression may take a few minutes before the download begins).

Note: You can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for process_output.log , you can type either process or .log

results-pane-search-file.png


By file(s) selection


Manually select the desired file(s) by clicking in the check box. Click on the Actions dropdown button and then click Download selected files

results-pane-manual-select.png


By direct download


Download individual files by clicking on the Download icon

results-pane-download-icon.png

The Files tab on the menu bar shows all files from all the jobs previously run from your account.

Go to the Files tab on the menu bar

files-tab-main.png

The options are:


Search for files


Type the file name or extension in the search bar. Click on Search. Click in the check box. Click on the Actions dropdown button and then click Download selected files

Note: You can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for all files surface_flows.csv, you can type either surface_flows or .csv

files-tab-search-file.png


Advanced search


Click on Advanced Search. Type the file extension in the search bar. You can look for multiple file extensions here. Click on Add after entering each file extension. Click on Search.

files-tab-advanced-search.png

Once the relevant files are displayed, click on the Check box next to Name. Click on the Actions dropdown button and then click Download selected files


Manually select files


Manually select the desired file(s) by clicking in the check box. Click on the Actions dropdown button and then click Download selected files

files-tab-manual-select.png

The Snapshot feature allows you to take periodic snapshots of the working directory for a run of your Rescale job, either for restart purposes or for additional review on your local machine.

When your job is running, the Status Tab will have a Live Tailing window. If you are running a job with multiple runs, such as a DOE, there may be multiple runs in the Runs column.

  • Select the run of your choice from the Runs column
  • Click on the Snapshot button to create a snapshot of the current working directory, including subdirectories

snapshot-status-page.png

  • All files are packaged into a single zip file. Once that zip file is ready, a download button will appear, as shown below. Click on Download

snapshot-popup.png

Snapshots that you create are also available for later download from the Results tab. If you created multiple snapshots over the course of the run, each one will be listed and labeled with the date and time it was taken.

snapshot-results.png

The "scp" linux command allows you to securely copy files from the compute cluster to the local workstation while a job is currently running.

The steps are:

  • Set your SSH Public Key
  • Connect to the Cluster
  • Determine the files you want to copy and the destination on the local workstation

Here is an example:

scp-ssh-cluster.png

Public key id_rsa.pub
Location of public key ~/.ssh/
Cluster Id xxxx_xXxxxx@99.99.99.99
Source Directory ~/work/
File to copy filetocopy.log
Destination /Users/johndoe/cfdruns/
  • In the terminal type, type the following for the above example
scp -i ~/.ssh/id_rsa.pub xxxx_xXxxxx@99.99.99.99:~/work/filetocopy.log      
/Users/johndoe/cfdruns/

The relevant files are copied to the local workstation.

The Rescale Transfer Manager is a native Windows application that can be used to download output files from jobs. This is a more robust and faster alternative to downloading large files through your browser.

At the moment, Rescale Transfer Manager is only available on Windows Operating Systems.

The steps are:

Setting up the Rescale Transfer Manager

  • At the top right in your Rescale account, click on your user email address, click on Account
  • Select API on the left-hand side of the page
  • Click Generate to create an API key. A 40-digit key should display. This will provision an API key for your account. You will only have to do this once (per account lifetime). A 40-digit key should display

Example

  • Press on Click to Install, and run the downloaded setup.exe file
  • An application install window will pop-up, click install, and click Run
  • The Rescale Transfer Manager Get Started window will display. In the drop down list select the right platform based on the region you live in
  • Paste the API key in the space provided and click OK

Example

Download all job output files

  • Go to the Results pane for the relevant job
  • Click on Download with Rescale Transfer Manager. If a warning message pops up, click on Open Rescale.TransferClient.UI

Example

  • A Browse for Folder window will open. Select a destination folder and click OK.

Example

Download a subset of the job output files

  • Note the job ID of the job that you want to download. The job ID is a series of upper and lowercase letters that is displayed in the URL . Paste the Job ID in the field

Example

  • ype the file type you want to download

Note that you can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for newfile.pdb , you can type either newfile or .pdb

  • Click on Download

Example

Download output files automatically when jobs complete

The Transfer Manager has a simple background download feature that can be used to monitor your jobs and automatically start downloading output files when the job finishes. This can be a useful feature to enable if you are submitting a number of jobs at the same time and don't want to manually download the results from each one individually.

  • Click on the gear icon in the upper right to open the Settings page
  • Enable Automatic Downloads by checking the Enabled box. This will reveal additional settings that control which jobs and files will be automatically downloaded

Example

  • Max job age (in days) specifies the oldest completed job that will be downloaded. In the screenshot above, all completed jobs that were created in the last 30 days will be downloaded
  • Destination indicates the directory that jobs will be saved to
  • Search Query will restrict downloads to files with specified values in the filename
  • Click on Save

Note that the Transfer Manager application must remain open for automatic downloads to work.