Rescale CLI

A command-line app for submitting jobs to Rescale

The Rescale Client App or Rescale Command Line Interface (CLI) is a Java-based client application for submitting and managing jobs to Rescale without having to explicitly know the Rescale API. Aside from providing a simple command line utility for the Rescale API, it also integrates the multi-threaded transfer library for uploading files directly to the Rescale Cloud Storage. This also handles any authentication related to uploading/downloading files from the Rescale Data Store. The App has several main functions:

  • Submit: submitting a job in the current working directory to Rescale
  • Status: obtaining the status of a running job on Rescale
  • Sync: syncing output files of jobs to your hard drive
  • Retrieve Info: retrieve metadata from the Rescale API

This section describes the steps needed to set up the Rescale CLI App on your local workstation. You will need to set up your API Key settings and install Java to download and use the Rescale CLI App.



Obtaining an API Key Token

In order to utilize the Rescale CLI App, an API key token must be generated to properly identify the user and associated account.

To create this API token, in the Rescale Web UI, click on the drop down menu associated with the user account name (usually your email address) in the upper right hand corner. Select Account from the drop down menu. Then from the navigation menu on the left choose API. This should bring you to: https://platform.rescale.com/user/settings/api-key/

From this page, click the Generate button to create a new API key. Make note of this string, but you can return to the page at anytime to retrieve it. This string is your private <api-token>.

API-CLI Download



Getting the Rescale CLI App

Currently, the Rescale CLI Java-based app is not supported on Java version 9 or 10. You can download Java 8 from the following link

On the Rescale API page, below your API token, the Rescale CLI Java-based app can also be downloaded.

Please move the downloaded file (rescale.jar) from the 'Downloads' folder into a directory in your path. For Linux and Mac OS X systems, we recommend putting the file in the /usr/local/bin/ directory. Also, make sure this directory is included in your $PATH environment variable. Otherwise when invoking CLI commands ensure that the full path to this file is specified.

Basic Submission Syntax

With your API token and local copy of the App, a job can be submitted using the following command:

java -jar /usr/local/bin/rescale.jar submit -p <api-token> -i <input-script>

For users from other platforms (KR, JP, EU), please refer to CLI FAQs for additional flag required while using Rescale CLI. The <input-script> contains the commands the user may want to run on Rescale. It can be any kind of .sh shell script that can be executed from the command line using the ./ prefix. Regardless of the type of script, the App will try to parse the script to minimize inadvertent issues related to environment setup and un-supported commands. The -p flag is used to read the Rescale API Token you created as shown here.



Simplifying CLI Commands

You can simplify the rescale commands by creating wrapper scripts. You can also set the RESCALE_API_KEY environment variable (2) or create a profile in the configuration file (3) to omit -p <api-token> from Rescale CLI commands. These additional steps will help the user in simplifying and using the Rescale CLI commands easily.


1. Creating a wrapper script

In particular, creating a simple wrapper script will reduce some potential sources of error when submitting a job.

The following is an example contents of a bash wrapper script:

#!/bin/bash 
java -jar /usr/local/bin/rescale.jar submit -p <api-token> -E -i $1

Replace the <api-token> with the hash string of your API key token. The script also assumes that the Rescale CLI App (rescale.jar) is located in /usr/local/bin, so modify accordingly. Because the script will contain your private <api-token> value, it should be stored in a directory in your $PATH with limited permissions such as /usr/local/bin. Also, ensure that the script has the proper permissions so that it can be executed:

chmod +x /usr/local/bin/rescale-submit

2. Setting the RESCALE_API_KEY environment variable

For using the Rescale CLI commands, the Rescale API key must be used everytime. Since the API Key is a long combination of letter, it will be helpful to create an environment variable and save it in your profile so that you can conviniently use the API variable rather than the actual key everytime you use the commands. You can follow these steps for creating the environment variable -

  • On your terminal, type either vi ~/.bashrc OR vi ~/.bash_profile and hit ENTER.
  • This opens up a terminal editor where you can type the following :
    export RESCALE_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • Once you finish typing, hit ESC first and then type :wq to save the changes.
  • You can test the variable by displaying the variable using echo $RESCALE_API_KEY. This will display the Rescale API key on the terminal.

3. Using profiles in a configuration file

By default, the CLI will look for the profile file in ~/.config/rescale/apiconfig. Create the file (vi ~/.config/rescale/apiconfig), and add the following contents:

[default]
apikey = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The CLI will use the [default] profile by default. A different named profile can be specified by using the --profile command line argument. It is also possible to use a different config file path by setting the RESCALE_CONFIG_FILE environment variable to the desired path.



Submitting a Test Job - Hello World!

Let's look at a very simple example. Again, this example assumes that you have set up a wrapper script called rescale-submit shown above.

To run a "Hello World!" job, from a shell terminal (such as in Linux, MacOS, or Cygwin) create a new run directory. Using the text editor of your choice, create a submit.sh input submission script containing:

#!/bin/bash 
#RESCALE_NAME="Hello World" 
#RESCALE_CORES=1 
echo "Hello World!" > hello_world
sleep 120
 
#Create some input files: 
echo "file 0" > file0
mkdir inputs
echo "file 1" > inputs/file1
echo "file 2" > inputs/file2

Note: When copying the above script, ensure that there are no spaces following each environment variable.

Submit the job:

rescale-submit submit.sh

This will spin up a new cluster and submit the jobs with the commands from the script. The user can now login to the Rescale Web UI to tail and monitor the job. The job list will show the job as: "Hello World"

Job Submitted

The status page will show the progress of the job and the files can be tailed from within the UI by clicking on the file:

Live Tailing

The log will look something like this:

Running SGE version of Rescale Client App
Executing Command.
Parsing Input Files
No existing files to include
Did not find explicitly specified analysis, checking implicitly selected analyses
No implicitly selected analyses, assuming user-included analysis
No core type specified by user. Using default core type 'Standard'
Zipping Files
Creating temporary encrypted zip at /Users/rescale/helloworld/input.zip
Finished writing encrypted file
Uploading Files
Uploading: /Users/rescale/helloworld/run.sh
Uploading: /Users/rescale/helloworld/input.zip
Job qMCOdb: Saved
Job qMCOdb: Submitting
Job qMCOdb: Starting polling cycle
Job qMCOdb: Status - Validated
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Executing
Job qMCOdb: Status - Completed
Job qMCOdb: Finished...
Job qMCOdb: Downloading files to output/
Downloading output/submit.sh
Downloading output/run.sh
Downloading output/process_output.log
Downloading output/inputs/file2
Downloading output/inputs/file1
Downloading output/hello_world
Downloading output/file0
Finished downloading files.

This section provides a list of various commands the Rescale CLI App supports. You can also go through the Rescale CLI Tutorial for additional information on using CLI commands. Please note the following -

  • If you have set up the API key according to methods 2 or 3 in the Simplifying CLI Commands page, you may omit [-p <api-token>] from the below commands.

  • If you wish to get help on Rescale CLI Commands, you can use the -h flag. Example -

    java -jar /usr/local/bin/rescale.jar -h
    
Rescale Cores/Software Information

You can obtain information about Rescale Core Types and the Softwares available using the following commands and flags :

COMMAND java -jar /usr/local/bin/rescale.jar list-info -<flag-type> [-p <api-token>]

FLAGS DESCRIPTION
-c Retrieves information about RESCALE core types
-a Retrieves information about softwares available


Job Management

Submit a job

Submits a job on the Rescale platform. You can login to the Rescale platform to check the submitted job.

COMMAND java -jar /usr/local/bin/rescale.jar submit [-p <api-token>] -i <input-script>

FLAGS DESCRIPTION
1) -q or --quiet Suppresses all logging output that would normally go to STDOUT. This is useful when piping the script output to another script.
2) -E or --end-to-end Runs the job end-to-end : uploading input files, submitting the job, monitoring for the results and downloading the results files.
3) -f <file1> <file2> ... <fileN> Filter output files to be downloaded after the job finishes. This flag can be used with submit only while running an end-to-end job.


Stop a job

Submits a request to cleanly shutdown the currently running job

COMMAND java -jar /usr/local/bin/rescale.jar stop [-p <api-token>] -j <job-id>


Delete a job

Submits a request to delete a completed job

COMMAND java -jar /usr/local/bin/rescale.jar delete [-p <api-token>] -j <job-id>


Check job status

Requests the specified job's status. Place in a loop to monitor continuously. It is recommended that an exponential backoff algorithm is used to monitor jobs that take long.

COMMAND java -jar /usr/local/bin/rescale.jar status [-p <api-token>] -j <job-id>



File transfer

Download files from a completed job

Downloads all the result files after the job run.

COMMAND java -jar /usr/local/bin/rescale.jar sync [-p <api-token>] -j <job-id>

FLAGS DESCRIPTION
-s <file1> <file2> ... <fileN> Output file filtering using "string pattern matching" searching.
(Since it happens on the server side, searching more robust
and faster - Recommended)
-f <file1> <file2> ... <fileN> Output file filtering using "glob" (Happens on the client side).
*is a wildcard that will match any string of characters and
? is a wildcard that will match any single character
--exclude <file1>,<file2>, ... ,<fileN> Excludes the specified files from downloading.
-n <job-id> To sync all jobs newer than a specific job. App will create
individual directories as - rescale_job_<job-ID>
-d <sync-time> For continuous syncing, the user may specify a
sync time. The time in seconds would determine
the delay time between two sync attempts


Download files from a running job

Downloads a specific file from a running job. It downloads a copy of the file (as it exists at that time) to the user's local directory in which they executed the command. Second, it would adds that same time-sensitive copy of the file to the job's output, whose name is formatted "download_{timestamp}_{original_filename}".

COMMAND java -jar /usr/local/bin/rescale.jar download-file [-p <api_key>] -j <job_id> -f <filename>

FLAGS DESCRIPTION
-f <file1> <file2> ... <fileN> Output file filtering using "string pattern matching"
(Happens on the client side). *is a wildcard that will
match any string of characters and ? is a wildcard
that will match any single character


Upload files to Rescale cloud storage

COMMAND java -jar /usr/local/bin/rescale.jar upload -p <api-token> -f <file1> ... <fileN>

FLAGS DESCRIPTION
-e To see an extended set of metadata during upload
--quiet To suppress non-json output from the command, if the CLI is being embedded in a larger script.


Persistant Clusters
FLAG DESCRIPTION
--p-cluster <cluster-id> This flag will enable the submission of additional jobs to a persistent cluster. The persistent cluster must be launched through the Web UI and already be running. The <cluster-id> value can obtained from the Web UI URL for the specific cluster (ie https://platform.rescale.com/clusters/<cluster-id>/status). Start by selecting the Clusters tab in the navigation bar, and then select the persistent cluster instant.
Note that this ID is not the <job-id>. Also, as with all persistent clusters, the user must terminate the cluster through the Web UI. The following image shows how to find the `: Cluster ID

This section shows different Rescale CLI command functionalities. It shows how to write a job submission script file, submit a job, check job status and shows different ways to download files. The tutorial is based on an LS-Dyna simulation example available from the Rescale resources page.

You can obtain the LS-DYNA simulation file (neon.refined.rev01.k) used in this example from the example problem.



Input Script and Job Submission

To submit a job that runs on a particular software (LS-DYNA in this case), you need to specify the software in your submit.sh file.

#!/bin/bash 
#RESCALE_NAME="LS-DYNA Neon Example" 
#RESCALE_ANALYSIS=ls_dyna 
#RESCALE_CORE_TYPE=Onyx 
#RESCALE_CORES=2 
#RESCALE_WALLTIME=1 
#USE_RESCALE_LICENSE 
ls-dyna -n $RESCALE_CORES_PER_SLOT -i neon.refined.rev01.k -p single

Note: When copying the above script, ensure that there are no spaces following each environment variable. You can find the flags for the input script in the Rescale CLI Commands section. Also, the command on the last line is the same as you would use on the Rescale platform; you can find the command on the Rescale Web UI or using the list-info command (please refer to the Rescale CLI Commands).

Assuming you have your Rescale API Key environment variable (refer here), run the job using the command:

java -jar /usr/local/bin/rescale.jar submit -i submit.sh

The log will look like this on your terminal:

Running SGE version of Rescale Client App
Executing Command.
Parsing Input Files
No existing files to include
No core type specified by user. Using default core type 'Nickel'
Found Analysis: ls_dyna
Zipping Files
Creating temporary encrypted zip at /Users/rescale/lsdyna_job/input.zip
Finished writing encrypted file
Uploading Files
Uploading: /Users/rescale/lsdyna_job/run.sh
Uploading run.sh:
##############################| 176B / 176B
Uploading: /Users/rescale/lsdyna_job/input.zip
Uploading input.zip:
##############################| 9.91MB / 9.91MB
Job: Saving Job
Job rqRXT: Saved
Job rqRXT: Submitting
Job rqRXT: --end-to-end flag not set, polling should be done manually.


Job Submission Files

The App will zip up all files in the current working directory into a temporary file called input.zip. It will then upload both run.sh and input.zip to Rescale and include them as input files to the job. It will also include any files listed in the #RESCALE_EXISTING_FILES line of the input script. For example, if your directory looks like this:

neon.refined.rev01.k
submit.sh

And you submit your job, the App will temporarily create the zip file, as well as parse the submit.sh script into run.sh. Your directory will look like this:

neon.refined.rev01.k
run.sh
submit.sh

The input.zip will include the following files - neon.refined.rev01.k, run.sh, submit.sh. The App will upload the input.zip file to Rescale along with the run.sh script. The App will then remove the zip file from the local file system and attempt to save/submit the job. If the job passes validation it will be saved and submitted to Rescale. The App will then start a polling cycle and wait for the status of the job to reach completed. At this point, the App will also print out a <job-id> which can be used for reference.



Checking Job Status

To check the status of the job running, type the following command by replacing rqRXT with your job ID -

java -jar /usr/local/bin/rescale.jar status -j rqRXT

The terminal will show the status of the message as shown -

java -jar /usr/local/bin/rescale.jar status -j rqRXT
Authenticated as <user>@rescale.com
The status of job rqRXT is Executing


Downloading a File During Job Run

To download a file when the job is running, we use the download-file CLI command as show below. You can download only one file at a time.

java -jar /usr/local/bin/rescale.jar download-file -j rqRXT -f shared/d3plot

Here, we are downloading the file d3plot from the run. The command looks for the file in the cluster working directory and hence the file path relative to the working directory must be specified. Here, the file d3plot is present in the shared directory. To obtain the relative path of the file, please check the live-tailing window on the Rescale platform as shown.

Relative File Path



Downloading Files After Job Has Completed

To download files after a job has completed, we use the sync CLI command. You can download all the files of the completed job as shown -

java -jar /usr/local/bin/rescale.jar sync -j rqRXT

The terminal will show the following log -

java -jar /usr/local/bin/rescale.jar sync -j rqRXT
Authenticated as shashank@rescale.com
Syncing output files for job vOUOeb
Job rqRXT: Downloading files to /Users/shashank/Documents/CLI/LS-Dyna-Example/rescale_job_rqRXT
Downloading /Users/shashank/Documents/CLI/LS-Dyna-Example/rescale_job_rqRXT/download_20180703_224544_d3plot
Downloading download_20180703_224544_d3plot:
Downloading /Users/shashank/Documents/CLI/LS-Dyna-Example/rescale_job_rqRXT/process_output.log
.
.
.
.
 

You can use output filtering using -f or -s flags to download only specific files. -f flag uses "glob" searching (On client side) while -s uses "string pattern" searching (On server side). Additionally, you can also use --exclude to exclude certain file from downloading.*is a wildcard that will match any string of characters and ? is a wildcard that will match any single character.

Example : The following command downloads all files starting with d3 while excluding d3hsp, d3dump01.0000 and d3dump01.0001 :

java -jar /usr/local/bin/rescale.jar sync -j rqRXT -f d3* --exclude d3hsp,d3dump01.0000,d3dump01.0000

For large jobs, the -s is recommended since it is on the server side.

Note -s does only string pattern searching and downloads all files with the given string pattern.

The Input Parser

The App uses the SGE input parser by default to parse information from the runtime script to be used as settings on Rescale, and to parse out any unwanted cluster-specific settings in the input submission script (not the wrapper script). For example, the user may have a script named submit.sh that looks like this:

#!/bin/bash 
#RESCALE_NAME="Hello World!" 
module load hello-world
source /home/rescale/environments/hello_world
/usr/lib64/openmpi/bin/mpirun echo "hello-world" > $(hostname)

The parser will try to make sure it parses out the cluster specific line module load... to make sure the script doesn't error on this line. It will then try to remove any commands that change the environment. In this case the lines after source will be parsed out so it doesn't interfere with the Rescale environment. Additionally, the parser will try to remove absolute references to executables/binaries as it is very likely that the executable will not reside in the same location on Rescale as on the local cluster. In this case, the reference to mpirun will be referenced through the PATH environment variable in the Rescale environment. The resulting script will then be saved to a file named run.sh and will look like this:

#!/bin/bash 
#RESCALE_NAME="Hello World!" 
#module load hello-world 
#source /home/rescale/environments/hello_world 
mpirun echo "hello-world" > $(hostname)

The Rescale analysis command will be set to ./run.sh. This script will be uploaded to Rescale along with the input files.



Rescale Environment Variables

The App supports the use of Rescale-specific settings environment variables settings inside the submit script to allow the user to configure their jobs exactly the way they want it. The currently available variables are listed below. If these variables are not included, the default value will be used.


#RESCALE_CORES=<integer>
{1|2|4|8|16|32|64|128|...}

The number of cores to use. Depending on the core type, the allowed values is a subset of the listed options. The default value for this option is the SGE #$ -pe value.


#RESCALE_CORE_TYPE=<string>
{Marble|Nickel|Onyx|Iron|Gold|Obsidian|...}

The Rescale core type to use. See the Commands section in this document for instructions on how to retrieve a list of valid values for this option.


#RESCALE_NAME=<string>

The name of your job. The default value for this option is the SGE #$ -N value.


#RESCALE_PROJECT_ID=<project name>

Use this when jobs are required to be assigned to a project as set up in the company platform settings.


#RESCALE_ANALYSIS=<string>
{abaqus|adina|aermod|ansys_cfx|ansys_fluent|...|user_included|...}

The analysis used. This should be set to an Analysis "code" value. The Commands section in this document contains instructions on how to retrieve a list of valid values for this option. The default value is any analysis name that can be parsed and is specified in the module or source lines of the script.


#RESCALE_ANALYSIS_VERSION=<string>

The specific version of the selected analysis to use. This is an optional setting. If omitted, the latest version will be used. If specified, this should be set to the "versionCode" value for a specific analysis version. The Commands section in this document contains instructions on how to retrieve a list of valid values for this option.


#RESCALE_EXISTING_FILES=<comma delimited list of strings>

For example: #RESCALE_EXISTING_FILES=<file1-ID>,<file2-ID>,<file3-ID>

This flag will include the listed file IDs if they are already stored on Rescale in this job. Obtaining file IDs is not straight-forward and the user would need access to the API. Please contact Rescale Support (support@rescale.com) for this. If the file IDs listed does not exist, the job will fail on validation. There is no default value for this option.


#USE_RESCALE_LICENSE

Use the Rescale provided licenses for the analysis code. If already existing licenses are to be used, those can be set up using environment variables, using the execution script.


#RESCALE_ENV_<varname>=<value>

Used to set license server information on the Rescale cluster. Replace with the name of the environment variable name and with the license server location. For example: #RESCALE_ENV_RLM_LICENSE=8112@license-proxy.rescale.com will create a RLM_LICENSE environment variable set to 8112@license-proxy.rescale.com on the cluster nodes.


#RESCALE_WALLTIME=<# of hours>

An optional value used to set the maximum number of hours that this job will be allowed to run. Jobs that exceed this number of hours will be terminated. If not specified, the default Max Job Hours value listed in your Rescale user account settings will be used.


#RESCALE_LOW_PRIORITY=<boolean>
{true, false}

An optional value used to run your job in on demand mode. With on demand enabled, jobs may initially enter a queue state until resources become available. Once this on demand job is running, it can be terminated and restarted at most one time. If this happens, jobs may take up to double the runtime. If not specified, the default value is false resulting in Instant mode.



Advanced SSH Settings

There are some additional advanced settings that can be used to enable SSH access into the master node of the Rescale cluster. Keep in mind that while this can be useful for examining and modifying the results of a job, it negatively impacts the reproducibility of the results should you ever clone and re-run the job in the future.

Currently, there is no way to retrieve the IP address of the master node from the CLI. You will need to use the Job status page on the Web UI to find this information.


#RESCALE_INBOUND_SSH_CIDR=<string>
for example: #RESCALE_INBOUND_SSH_CIDR=50.123.22.112/32

Used to define an IP range that will be allowed inbound SSH access to the master node of the Rescale cluster. If this value is omitted, the default value listed in your Rescale user account settings will be used.


#RESCALE_PUBLIC_KEY=<string>
for example: #RESCALE_PUBLIC_KEY=ssh-rsa AAAA...

Used to set the public key that will be added to the authorized_keys file of the master node on the Rescale cluster. This will allow you to SSH into the master node of the Rescale cluster with the associated private key. If this value is omitted, the default value listed in your Rescale user account settings will be used.




Example Scripts

An example script that will both work on SGE and Rescale could look something like this (the module line will be commented out, and the absolute reference to mpirun will be removed):

#!/bin/bash 
#RESCALE_NAME="Converge Sample" 
#RESCALE_CORES=32 
#RESCALE_CORE_TYPE=Nickel 
#RESCALE_ANALYSIS=converge_open_mpi 
module load converge
/usr/bin/mpirun -np 32 converge-2.1.0

An example script that will only work on Rescale could look something like below. The command for the command line can be found in the Analysis Code Table (separate attachment).

#!/bin/bash 
#RESCALE_NAME="Converge Sample" 
#RESCALE_CORES=32 
#RESCALE_CORE_TYPE=Nickel 
#RESCALE_ANALYSIS=converge_open_mpi 
converge-mpi -n all -v 2.1
CLI Flag for specifying different regional platforms

Rescale CLI defaults to the US platform platform.rescale.com. If you are a user on any of the other global platforms (EU, JP, etc.), please specify your regional platform using the -X flag in your CLI command. For example:

java -jar /usr/local/bin/rescale.jar -X https://eu.rescale.com submit ...


Debugging Issues with CLI

In addition, there are a few advanced Java system properties that can be specified:

rescale.loglevel (TRACE, DEBUG, INFO, WARNING, ERROR, OFF)
https.proxyHost
https.proxyPort
https.proxyUser
https.proxyPassword

For example, to run with DEBUG logging turned on and use proxy.com:8888 as a proxy server, the following command can be used:

java -Drescale.loglevel=DEBUG -Dhttps.proxyHost=proxy.com -Dhttps.proxyPort=8888 -jar /usr/local/bin/rescale.jar status -p <api-token> -j <job-id>

Note: There is no spacing between the -D and the name of the system property. In addition, the -D switches should also be specified before the -jar option.

If the proxy requires Windows authentication, the https.proxyUser and https.proxyPassword system properties can be set. The https.proxyUser value should be set to DOMAIN\username. The https.proxyPassword property can be set to the Windows password for the user. If you do not want to embed the password in the command line, this property can be omitted and the CLI will prompt you to enter your password in the console when it launches.



Upload logs, metadata and supressing non-json data

During the uploading, after each file is successfully uploaded there will be a standard output indicating so. So the user can also redirect the output into a log file so all the successful uploads will be recorded.

java -jar /usr/local/bin/rescale.jar upload -p <api-token> -f <file1> ... <fileN> > upload_20141201.log

By default, the upload command will return a limited set of metadata for the uploaded file. To see an extended set of metadata, use the -e flag in the upload call:

java -jar /usr/local/bin/rescale.jar upload -p <api-token> -f <file1> ... <fileN> -e

Finally, if the CLI is being embedded in a larger script, then it is useful to use the --quiet flag to suppress non-json output from the command. This makes the parsing the response easier:

java -jar /usr/local/bin/rescale.jar --quiet upload -p <api-token> -f <file1> ... <fileN> -e