File Transfer to Local Workstation

Rescale platform offers several ways to transfer files to your local workstation

After your job has finished, the Results pane displays all the files associated with the job.

Go to the Results pane for your job

results-pane-main.png

On the Results pane, the options to download files are:


Download entire job


Click on Download Job to download a zip file containing all files associated with the run, including input and output files

results-pane-download-job.png


Search for files


Type the file name or extension in the search bar. Click on Search. Click in the check box. Click on the Actions dropdown button and then click Download selected files

Note: If a single file is selected, the file itself will be downloaded. If more than one file is selected, the files will be compressed first and then download (the compression may take a few minutes before the download begins).

Note: You can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for process_output.log , you can type either process or .log

results-pane-search-file.png


By file(s) selection


Manually select the desired file(s) by clicking in the check box. Click on the Actions dropdown button and then click Download selected files

results-pane-manual-select.png


By direct download


Download individual files by clicking on the Download icon

results-pane-download-icon.png

The Files tab on the menu bar shows all files from all the jobs previously run from your account.

Go to the Files tab on the menu bar

files-tab-main.png

The options are:


Search for files


Type the file name or extension in the search bar. Click on Search. Click in the check box. Click on the Actions dropdown button and then click Download selected files

Note: You can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for all files surface_flows.csv, you can type either surface_flows or .csv

files-tab-search-file.png


Advanced search


Click on Advanced Search. Type the file extension in the search bar. You can look for multiple file extensions here. Click on Add after entering each file extension. Click on Search.

files-tab-advanced-search.png

Once the relevant files are displayed, click on the Check box next to Name. Click on the Actions dropdown button and then click Download selected files


Manually select files


Manually select the desired file(s) by clicking in the check box. Click on the Actions dropdown button and then click Download selected files

files-tab-manual-select.png

The Snapshot feature allows you to take periodic snapshots of the working directory for a run of your Rescale job, either for restart purposes or for additional review on your local machine.

When your job is running, the Status Tab will have a Live Tailing window. If you are running a job with multiple runs, such as a DOE, there may be multiple runs in the Runs column.

  • Select the run of your choice from the Runs column
  • Click on the Snapshot button to create a snapshot of the current working directory, including subdirectories

snapshot-status-page.png

  • All files are packaged into a single zip file. Once that zip file is ready, a download button will appear, as shown below. Click on Download

snapshot-popup.png

Snapshots that you create are also available for later download from the Results tab. If you created multiple snapshots over the course of the run, each one will be listed and labeled with the date and time it was taken.

snapshot-results.png

The "scp" linux command allows you to securely copy files from the compute cluster to the local workstation while a job is currently running.

The steps are:

  • Set your SSH Public Key
  • Determine the files you want to copy and the destination on the local workstation. The job files are located in the ~/work/ directory. You can also connect to your cluster using Rescale's In-Browser SSH or Manual SSH to verify the location of the file.
  • Use the following command line on your local terminal to transfer files :
scp -i ~/.ssh/<private key> <source> <destination>
  • To access the cluster directory, we must specify the cluster ID, which can be found as shown:

scp-ssh-cluster.png

  • Here is an example for transferring files from the cluster to your local workstation
Private key id_rsa
Location of SSH key ~/.ssh/
Cluster ID xxxx_xXxxxx@99.99.99.99
Source Directory ~/work/
File to copy process_output.log
Destination /Users/johndoe/cfdruns/
  • In the terminal type, type the following for the above example
scp -i ~/.ssh/id_rsa xxxx_xXxxxx@99.99.99.99:~/work/process_output.log /Users/johndoe/cfdruns/
  • The file process_output.log is transfereed from the cluster to the folder cfdruns.
  • You can transfer multiple files with similar name using the asterisk symbol *. For example, to transfer files - velocity-contour-0, velocity-contour-1, velocity-contour-2, velocity-contour-3, velocity-contour-4, the scp command will look like -
scp -i ~/.ssh/id_rsa xxxx_xXxxxx@99.99.99.99:~/work/velocity-contour-* /Users/johndoe/cfdruns/
  • If you are copying a file from your local workstation to the cluster, the scp command will look like this -
scp -i ~/.ssh/id_rsa /Users/johndoe/cfdruns/FileToTransfer.txt xxxx_xXxxxx@99.99.99.99:~/work/
  • The above command will transfer the file FileToTransfer.txt from the local workstation to the work directory of the cluster.

WinSCP is a popular file transfer client for Microsoft Windows. You can download WinSCP from here. Please follow these steps to transfer using WinSCP -

  • Create your public/private key pair and copy them to a folder you can access in your local workstation. For information on creating SSH keys, please refer to Manual SSH > Windows user > Generating your SSH keys in Connecting to your cluster resource page.
WinSCP Keys
  • Open WinSCP and fill out the login information from Rescale as shown below. Select File protocol as SFTP, Host name with the IP XX:XXX:XX:XX and User name with the job ID uprod_xxXxX. The IP and job ID can be found on the Rescale platform as shown below
WinSCP Login
  • Click on Advanced > Advanced... and then click on Authentication in the tree list on the left. Under the private key file field, browse for the newly created private key.
WinSCP Advanced
  • Accept the changes and click on Login. Your public key should now be authenticated and you will be connected to the cluster as shown below.
WinSCP Authernticating
  • You can transfer files between your local workstation (on the left) and the cluster (on the right).
WinSCP Window

The Rescale Transfer Manager is a native Windows application that can be used to download output files from jobs. This is a more robust and faster alternative to downloading large files through your browser.

At the moment, Rescale Transfer Manager is only available on Windows Operating Systems.

The steps are:

Setting up the Rescale Transfer Manager

  • At the top right in your Rescale account, click on your user email address, click on Account
  • Select API on the left-hand side of the page
  • Click Generate to create an API key. A 40-digit key should display. This will provision an API key for your account. You will only have to do this once (per account lifetime). A 40-digit key should display

Example

  • Press on Click to Install, and run the downloaded setup.exe file
  • An application install window will pop-up, click install, and click Run
  • The Rescale Transfer Manager Get Started window will display. In the drop down list select the right platform based on the region you live in
  • Paste the API key in the space provided and click OK

Example

Download all job output files

  • Go to the Results pane for the relevant job
  • Click on Download with Rescale Transfer Manager. If a warning message pops up, click on Open Rescale.TransferClient.UI

Example

  • A Browse for Folder window will open. Select a destination folder and click OK.

Example

Download a subset of the job output files

  • Note the job ID of the job that you want to download. The job ID is a series of upper and lowercase letters that is displayed in the URL . Paste the Job ID in the field

Example

  • ype the file type you want to download

Note that you can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for newfile.pdb , you can type either newfile or .pdb

  • Click on Download

Example

Download output files automatically when jobs complete

The Transfer Manager has a simple background download feature that can be used to monitor your jobs and automatically start downloading output files when the job finishes. This can be a useful feature to enable if you are submitting a number of jobs at the same time and don't want to manually download the results from each one individually.

  • Click on the gear icon in the upper right to open the Settings page
  • Enable Automatic Downloads by checking the Enabled box. This will reveal additional settings that control which jobs and files will be automatically downloaded

Example

  • Max job age (in days) specifies the oldest completed job that will be downloaded. In the screenshot above, all completed jobs that were created in the last 30 days will be downloaded
  • Destination indicates the directory that jobs will be saved to
  • Search Query will restrict downloads to files with specified values in the filename
  • Click on Save

Note that the Transfer Manager application must remain open for automatic downloads to work.