Rescale CLI

A command-line app for submitting jobs to Rescale

The Rescale Client App ("The App") or Rescale Command Line Interface (CLI) is a Java-based client application for submitting jobs to Rescale without having to explicitly know the Rescale API. Aside from providing a simple command line utility for the Rescale API, it also integrates the multi-threaded transfer library for uploading files directly to the Rescale Data Store. This also handles any authentication related to uploading/downloading files from the Rescale Data Store. The App has several main functions:

  • Submit: submitting a job in the current working directory to Rescale
  • Status: obtaining the status of a running job on Rescale
  • Sync: syncing output files of jobs to your hard drive
  • Retrieve Info: Retrieve metadata from the Rescale API

Getting an API Key Token

First, in order to utilize the Rescale CLI App, an API key token must be generated in order to properly identify the user and associated account.

To create this API token, in the Rescale Web UI, click on the drop down menu associated with the user account name (usually your email address) in the upper right hand corner. Select Account from the drop down menu. Then from the navigation menu on the left choose API. This should bring you to:

From this page, click the Generate button to create a new API key. Make note of this string, but you can return to the page at anytime to retrieve it. This string is your private <api-token>.

API-CLI Download

Getting the Rescale CLI App

On the same page, below your API token, the Rescale CLI Java-based app can also be downloaded.

Please move the downloaded file from the 'Downloads' folder into a directory in your path. For Linux and Mac OS X systems, we recommend putting the file in the /usr/local/bin/ directory. Also, make sure this directory is included in your $PATH environment variable. Otherwise when invoking CLI commands ensure that the full path to this file is specified.

Basic Submission Syntax

With your API token and local copy of the App, a job can be submitted using the following command:

java -jar rescale.jar submit -p <api-token> -i <input-script>

The <input-script> contains the commands the user may want to run on Rescale. It can be any kind of .sh shell script that can be executed from the command line using the ./ prefix. The script may be a long submit script or a simple one-liner script. Regardless of the type of script, the App will try to parse the script to minimize inadvertent issues related to environment setup and un-supported commands.

Creating a Wrapper Script

We recommend you review the CLI FAQ page on how to simplify this command as well as implementing the API key integration. In particular, creating a simple wrapper script will reduce some potential sources of error when submitting a job.

The following is an example contents of a bash wrapper script:

java -jar /usr/local/bin/rescale.jar submit -p <api-token> -E -i $1

Replace the <api-token> with the hash string of your API key token. The script also assumes that the Rescale CLI App (rescale.jar) is located in /usr/local/bin, so modify accordingly. Because the script will contain your private <api-token> value, it should be stored in a directory in your $PATH with limited permissions such as ~/bin. Also, ensure that the script has the proper permissions so that that it can be executed:

chmod +x ~/bin/rescale-submit

For the following examples, we will assume that a wrapper script named rescale-submit exists and is available in your path.

Let's look at a very simple example. Again, this example assumes that you have set up a wrapper script called rescale-submit show above (or refer to the CLI FAQ).

To run a "Hello World!" job, from a shell terminal (such as in Linux, MacOS, or Cygwin) create a new run directory. Using the text editor of your choice, create a input submission script containing:

#RESCALE_NAME="Hello World" 
echo "Hello World!" > hello_world
sleep 120
#Create some input files: 
echo "file 0" > file0
mkdir inputs
echo "file 1" > inputs/file1
echo "file 2" > inputs/file2

Note: When copying the above script, ensure that there are no spaces following each environment variable.

Submit the job:


This will spin up a new cluster and submit the jobs with the commands from the script. The user can now login to the Rescale Web UI to tail and monitor the job. The job list will show the job as: "Hello World"

Job Submitted

The status page will show the progress of the job and the files can be tailed from within the UI by clicking on the file:

Live Tailing

The log will look something like this:

Running SGE version of Rescale Client App
Executing Command.
Parsing Input Files
No existing files to include
Did not find explicitly specified analysis, checking implicitly selected analyses
No implicitly selected analyses, assuming user-included analysis
No core type specified by user. Using default core type 'Standard'
Zipping Files
Creating temporary encrypted zip at /Users/rescale/helloworld/
Finished writing encrypted file
Uploading Files
Uploading: /Users/rescale/helloworld/
Uploading: /Users/rescale/helloworld/
Job qMCOdb: Saved
Job qMCOdb: Submitting
Job qMCOdb: Starting polling cycle
Job qMCOdb: Status - Validated
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Completed
Job qMCOdb: Finished...
Job qMCOdb: Downloading files to output/
Downloading output/
Downloading output/
Downloading output/process_output.log
Downloading output/inputs/file2
Downloading output/inputs/file1
Downloading output/hello_world
Downloading output/file0
Finished downloading files.

The App will zip up all files in the current working directory into a temporary file called It will then upload both and to Rescale and include them as input files to the job. It will also include any files listed in the #RESCALE_EXISTING_FILES line of the input script. For example, if your directory looks like this:


And you submit your job using rescale-submit the App will temporarily create the zip file, as well as parse the script into Your directory will look like this:


The App will upload the file to Rescale along with the script. The App will then remove the zip file from the local file system and attempt to save/submit the job. If the job passes validation it will be saved and submitted to Rescale. The App will then start a polling cycle and wait for the status of the job to reach completed. At this point, the App will also print out a which can be used for reference.

Once the job has completed, it will download all the files in the remote Rescale working directory to the local file system in a directory named output. Your working directory may look something like this:


The App will exit once all the output files have been downloaded. Multiple jobs can also be submitted to a persistent cluster.

To submit a job that runs on a particular software (LS-DYNA in this case), you need to specify the software in your file.

ls-dyna -n $RESCALE_CORES_PER_SLOT -i neon.refined.rev01.k -p single

Note: When copying the above script, ensure that there are no spaces following each environment variable. You can find the flags for the input script in the CLI FAQ. Also, the command on the last line is the same as you would use on the Rescale platform; you can find the command on the Rescale Web UI or using the list-info command (please refer to the CLI FAQ).

You can obtain the LS-DYNA simulation file (neon.refined.rev01.k) used in this example from an example problem.

Assuming you have the wrapper script set up (refer here), run the job using the command:


The log will look something like this:

Running SGE version of Rescale Client App
Executing Command.
Parsing Input Files
No existing files to include
No core type specified by user. Using default core type 'Nickel'
Found Analysis: ls_dyna
Zipping Files
Creating temporary encrypted zip at /Users/rescale/lsdyna_job/
Finished writing encrypted file
Uploading Files
Uploading: /Users/rescale/lsdyna_job/
##############################| 176B / 176B
Uploading: /Users/rescale/lsdyna_job/
##############################| 9.91MB / 9.91MB
Job: Saving Job
Job npnLT: Saved
Job npnLT: Submitting
Job npnLT: Starting polling cycle
Job npnLT: Status - Validated
Job npnLT: Status - Executing
Job npnLT: Status - Completed
Job npnLT: Finished...
Job npnLT: Downloading files to /Users/rescale/lsdyna_job/output
Finished downloading files.

To obtain the status of a job you can navigate through the Web UI to the job:<job-id>/ or you can also query it using the App with the status command. After a job is submitted, once the case is uploaded and saved, a unique <job-id> should be created. The standard output should look something like this:

2017-09-22 15:41:22,148 - Job: Saving Job
2017-09-22 15:41:22,586 - Job BMHxT: Saved
2017-09-22 15:41:22,586 - Job BMHxT: Submitting
2017-09-22 15:41:23,486 - Job BMHxT: Starting polling cycle

In this example the <job-id> is BMHxT. Using your appropriate <job-id>, you can query the job status with the following command:

java -jar rescale.jar status -p <api-token> -j <job-id>

*Note that this is using the status command and not the submit command, so the submission wrapper script will not work here without modification.

Command Line Commands

The App supports the following commands:

"status" requests the specified job's status. Place in a loop to monitor continuously. It is recommended that an exponential backoff algorithm is used to monitor jobs that take longer.

java -jar /usr/local/bin/rescale.jar status [-p <api-token>] -j <job-id>

"stop" submits a request to cleanly shutdown the currently running job.

java -jar /usr/local/bin/rescale.jar stop [-p <api-token>] -j <job-id>

"sync" downloads the results files.

java -jar /usr/local/bin/rescale.jar sync [-p <api-token>] -j <job-id>

"list-info" retrieves useful metadata from the Rescale API.

Core types (-c):

java -jar /usr/local/bin/rescale.jar list-info -c [-p <api-token>]

Software available (-a):

java -jar /usr/local/bin/rescale.jar list-info -a [-p <api-token>]

Please find a list of additional command line options/flags on the CLI FAQ page. Also, if you have set up the API key according to methods 2 or 3 in the CLI FAQ, you may omit [-p <api-token>] from the above commands.

The Input Parser

The App uses the SGE input parser by default to parse information from the runtime script to be used as settings on Rescale, and to parse out any unwanted cluster-specific settings in the input submission script (not the wrapper script). For example, the user may have a script named that looks like this:

#RESCALE_NAME="Hello World!" 
module load hello-world
source /home/rescale/environments/hello_world
/usr/lib64/openmpi/bin/mpirun echo "hello-world" > $(hostname)

The parser will try to make sure it parses out the cluster specific line module load... to make sure the script doesn't error on this line. It will then try to remove any commands that change the environment. In this case the lines after source will be parsed out so it doesn't interfere with the Rescale environment. Additionally, the parser will try to remove absolute references to executables/binaries as it is very likely that the executable will not reside in the same location on Rescale as on the local cluster. In this case, the reference to mpirun will be referenced through the PATH environment variable in the Rescale environment. The resulting script will then be saved to a file named and will look like this:

#RESCALE_NAME="Hello World!" 
#module load hello-world 
#source /home/rescale/environments/hello_world 
mpirun echo "hello-world" > $(hostname)

The Rescale analysis command will be set to ./ This script will be uploaded to Rescale along with the input files.

Rescale Environment Variables

The App supports the use of Rescale-specific settings environment variables settings inside the submit script to allow the user to configure their jobs exactly the way they want it. The currently available variables are listed below. If these variables are not included, the default value will be used.


The number of cores to use. Depending on the core type, the allowed values is a subset of the listed options. The default value for this option is the SGE #$ -pe value.


The Rescale core type to use. See the Metadata section in this document for instructions on how to retrieve a list of valid values for this option.


The name of your job. The default value for this option is the SGE #$ -N value.

#RESCALE_PROJECT_ID=<project name>

Use this when jobs are required to be assigned to a project as set up in the company platform settings.


The analysis used. This should be set to an Analysis "code" value. The Metadata section in this document contains instructions on how to retrieve a list of valid values for this option. The default value is any analysis name that can be parsed and is specified in the module or source lines of the script.


The specific version of the selected analysis to use. This is an optional setting. If omitted, the latest version will be used. If specified, this should be set to the "versionCode" value for a specific analysis version. The Metadata section in this document contains instructions on how to retrieve a list of valid values for this option.

#RESCALE_EXISTING_FILES=<comma delimited list of strings>

For example: #RESCALE_EXISTING_FILES=file1.dat,file2.dat,file3.dat

This flag will include the listed files if they are already stored on Rescale in this job. If the file(s) listed does not exist, the job will fail on validation. There is no default value for this option.


Use the Rescale provided licenses for the analysis code. If already existing licenses are to be used, those can be set up using environment variables, using the execution script.


Used to set license server information on the Rescale cluster. Replace with the name of the environment variable name and with the license server location. For example: will create a RLM_LICENSE environment variable set to on the cluster nodes.

#RESCALE_WALLTIME=<# of hours>

An optional value used to set the maximum number of hours that this job will be allowed to run. Jobs that exceed this number of hours will be terminated. If not specified, the default Max Job Hours value listed in your Rescale user account settings will be used.

{true, false}

An optional value used to run your job in on demand mode. With on demand enabled, jobs may initially enter a queue state until resources become available. Once this on demand job is running, it can be terminated and restarted at most one time. If this happens, jobs may take up to double the runtime. If not specified, the default value is false resulting in Instant mode.

Advanced Settings

There are some additional advanced settings that can be used to enable SSH access into the master node of the Rescale cluster. Keep in mind that while this can be useful for examining and modifying the results of a job, it negatively impacts the reproducibility of the results should you ever clone and re-run the job in the future.

Currently, there is no way to retrieve the IP address of the master node from the CLI. You will need to use the Job status page on the Web UI to find this information.


Used to define an IP range that will be allowed inbound SSH access to the master node of the Rescale cluster. If this value is omitted, the default value listed in your Rescale user account settings will be used.

for example: #RESCALE_PUBLIC_KEY=ssh-rsa AAAA...

Used to set the public key that will be added to the authorized_keys file of the master node on the Rescale cluster. This will allow you to SSH into the master node of the Rescale cluster with the associated private key. If this value is omitted, the default value listed in your Rescale user account settings will be used.

Example Scripts

An example script that will both work on SGE and Rescale could look something like this (the module line will be commented out, and the absolute reference to mpirun will be removed):

#RESCALE_NAME="Converge Sample" 
module load converge
/usr/bin/mpirun -np 32 converge-2.1.0

An example script that will only work on Rescale could look something like below. The command for the command line can be found in the Analysis Code Table (separate attachment).

#RESCALE_NAME="Converge Sample" 
converge-mpi -n all -v 2.1


There are several Rescale-specific flags that need to be filled out when submitting a job. The following metadata retrieval commands can be used to retrieve a JSON blob containing the data to use.

The following command will retrieve a list of core types:

java -jar /usr/local/bin/rescale.jar list-info -c -p <api-token>

The value listed for the name or code can be used in the RESCALE_CORE_TYPE flag. Note that the returned list includes deprecated types which can change over time so the user should make sure that the isDeprecated value is set to false for the desired core type.

The following command will retrieve a list of analyses and their versions:

java -jar /usr/local/bin/rescale.jar list-info -a -p <api-token>

Each item in the returned list represents an analysis. The value listed for "code" can be used in the RESCALE_ANALYSIS option. Within every analysis is a list of versions. The "versionCode" field can be used in the RESCALE_ANALYSIS_VERSION option.

Note: If you have set the API key according to methods 2 or 3 in the FAQ Section, you may omit [-p <api-token>] from the above commands.

--p-cluster Flag

If a persistent cluster, created through the Web UI, is already running, additional jobs may be submitted to this cluster using --p-cluster <cluster-id> flag.

java -jar /usr/local/bin/rescale.jar submit -p <api-token> -E -i --p-cluster <cluster-id>

If you have the rescale-submit wrapper script properly set up (refer to CLI FAQ), this can be simplified to:

rescale-submit --p-cluster <cluster-id>

Please refer to the CLI FAQ Section on runtime flags for more information about this and other flags. Please note that the <cluster-id> is found in the cluster process URL and which should be in the form:<cluster-id>/status). Note that this ID is not the same as a <job-id>. To find it, go to the Clusters tab in the navigation bar, and then select the specific persistent cluster process. The following image shows how to find the <cluster-id>:


If a job is successfully submitted to a running cluster, the cluster status page should show the additional job submitted to its process queue. However, by default, the jobs will not be managed by a scheduler, so multiple jobs will run simultaneously if submitted at the same time. Also, as with all persistent clusters, the user must terminate the cluster through the Web UI when jobs are completed.

Syncing (Downloading)

The user may want to sync remote output files with a local file system. One of the reasons the user may want to sync output files through the App is not having to manually go through jobs and downloading the files through the browser. There is a very basic implementation of syncing files built into the App which stores sync states in a .rescale file.

The sync commands must be run from a file system with enough disk space to accommodate the downloaded files.

If the user just wants to sync a single job the user would then run:

java -jar /usr/local/bin/rescale.jar sync -p <api-token> -j <job-id>

Also, the user may want to sync all jobs newer than a specific job, the user would then use the command:

java -jar /usr/local/bin/rescale.jar sync -p <api-token> -n <job-id>

For continuous syncing, the user may specify a sync time. The time in seconds would determine the delay time between two sync attempts:

java -jar /usr/local/bin/rescale.jar sync -p <api-token> -d <sync-time>

Given the working directory, the job output files will be synced to individual directories for each job. For example, the user may have the following jobs in order of creation date with the newest first: zzzZZZ, yyyYYY, xxxXXX, cccCCC, bbbBBB, aaaAAA.

If the user runs the command

java -jar /usr/local/bin/rescale.jar sync -p <api-token> -n xxxXXX

the App will create the following directories: rescale_job_zzzZZZ, rescale_job_yyyYYY, rescale_job_xxxXXX and sync the output files inside these directories. If the syncing of zzzZZZ, and yyyYYY succeed, but the syncing of xxxXXX fails, the next time the user runs the same command the App will not try to sync zzzZZZ and yyyYYY again, but will try to sync xxxXXX again.

The sync state files are saved in the respective output file directories, so if the user tries to run the sync command from a different working directory it will re-sync the files into that directory.

Output File Filtering

Both the submit and sync commands take an optional -f flag which can be used to restrict the output files that are downloaded to the ones that match the specified file name filters. Here are a couple of usage examples:

java -jar rescale.jar submit -p <api-token> -i <input-script> -E -f *.out process_output.log
java -jar rescale.jar sync -p <api-token> -j <job-id> -f image?.png

In the submit command, the file filtering option is only applicable when running an end-to-end job.

One or more file name matchers may be specified after the -f argument. Matchers are separated by spaces. * is a wildcard that will match any string of characters and ? is a wildcard that will match any single character.

Note that all matching is done against just the filename and does not match against the full path of the output file.


The user may want to upload one or several files from local file system to Rescale's cloud storage using the following command:

java -jar /usr/local/bin/rescale.jar upload -p <api-token> -f <file1> ... <fileN>

This command will check if all the files specified are actually files or not, and will stop if any of them is not. Note that the cli tool doesn't support uploading a directory yet.

During the uploading, after each file is successfully uploaded there will be a standard output indicating so. So the user can also redirect the output into a log file so all the successful uploads will be recorded.

java -jar /usr/local/bin/rescale.jar upload -p <api-token> -f <file1> ... <fileN> > upload_20141201.log

By default, the upload command will return a limited set of metadata for the uploaded file. To see an extended set of metadata, use the -e flag in the upload call:

java -jar /usr/local/bin/rescale.jar upload -p <api-token> -f <file1> ... <fileN> -e

Finally, if the CLI is being embedded in a larger script, then it is useful to use the --quiet flag to suppress non-json output from the command. This makes the parsing the response easier:

java -jar /usr/local/bin/rescale.jar --quiet upload -p <api-token> -f <file1> ... <fileN> -e