General FAQs

Updated: September 16, 2022

The Command string can specify any command or set of commands that you need to run the software you have selected. Commands that you specify may include scripts that you have uploaded as Input Files for the job. Multiple commands should be separated by semi-colons. Your Command string or script may call your chosen software multiple times. Using a script allows you to specify a more complicated workflow.

This Basic Job Example show a simple example using a script in the command line. This STAR-CCM+ Example calls the simulation software starccm+ directly. This LS-DYNA Example uses the Rescale ls-dyna wrapper script to simplify the command line for launching LS-DYNA.

Command Template contains suggestions to help you form your Command string and run the software you have selected. Templates may be used to invoke:

  • The Standard or serial version of an application
  • The SMP command
  • The MPI command

Codes that support hybrid parallelization can use the settings from the MPI Command Template.

The Command Template itself consists of either:

a) One of the standard commands that is used to run the software you have selected. It also typically includes the command line options that dictate the number of threads or processes to be used. The Comsol Multiphysics template is an example,

comsol -clustersimple -f $HOME/machinefile batch -inputfile <input-file>

b) A wrapper script command. Wrapper scripts are provided by Rescale to simplify the task of running the software you have selected in a way that helps you make efficient use of the hardware you have chosen. An example of a wrapper script template is LS-DYNA,

ls-dyna -n <mpi-ranks> -s <smp-ranks> -i <input> -p <precision> -a <other args>

We use the < > notation to show fields that you would typically be expected to replace with a real value before submitting a job. Make sure to remove the brackets once you have entered the value to avoid errors when running the command.

You do not always need to use all of the fields. If one is irrelevant to your situation, you can delete it. For instance, a code may support hybrid mode, but if you plan to use MPI alone, you can omit the value.

On Rescale, if the standard command line for a software package is particularly complex, we provide a wrapper script to allow you to access the correct executable more easily. One example of this is LS-DYNA, where the name of the executable you need to call depends on the software version, whether you choose to use single or double precision, and whether you use SMP, hybrid or MPP parallelization. Each wrapper script typically takes a number of simple command line options intended to help you call the exact executable you need and to utilize the hardware you have selected.

Use of wrapper scripts is entirely optional. Additionally, using the same executable command from a local run will also work on Rescale. If you want, you can log in to a cluster and view a wrapper script’s content by following the instructions at Connecting to Your Cluster. View your software’s wrapper script source by using the which command to locate the wrapper script, and the cat command to concatenate the wrapper script file’s contents.

These are presented in the each job’s Software Settings page. The Command Template presents the basic options that are available for StandardSMP and MPI modes where applicable.

Yes, you can. Most of the time, the executable you will need will already be on the $PATH environment variable when your command line or script is run. If necessary, you can login to a cluster to locate an executable not specified in that software’s Command Template, by following the instructions at Connecting to Your Cluster, or contact us directly at Rescale Support.

At runtime, host and machine files in the most common formats are located in the $HOME folder of your workspace. If your command line needs a host file as input data, you can specify it by name along with the $HOME prefix. For instance:

--hostfile $HOME/machinefile.openmpi

The **Command Template” for the software you have selected will normally refer to the exact hostfile you will need. If you list the contents of $HOME you will find the following host files:

hosts, mpd.hosts, machinefile, machinefile.gpu, machinefile.openmpi, hosts,rhosts, mpd.hosts.string, PCF.xml`

Some software requires a special file or format for host information. Wherever possible, Rescale creates environment variables and/or specific files as necessary for such software. For example, when you run Abaqus, Rescale places the host information in the correct format in the abaqus_v6.env file for you. When you run ANSYS, the $MACHINES environment variable contains the hosts information in the necessary format.

Sometimes software requires that the full path to a host (or other) file be placed inside one of your input files. On Rescale, you won’t know what the path will be in advance, but you will have access to environment variables representing it. The Shipflow example shows how you might use a sed command to substitute the string

parallel( nprocesses=2, nthreads=1, hostfile="$HOME/machinefile.openmpi" )

in the file hamb_def with its replacement

parallel( nprocesses=2, nthreads=1, hostfile="/enc/uprod_mtmHV/machinefile.openmpi" )

at runtime.

No. GPUs have hundreds of cores. Our pricing for a cluster with GPUs is based on the number of associated CPU cores on the cluster and not the number of GPU cores. A Rescale cluster with GPUs consists of computing nodes with multiple CPUs (typically 8) and one or more GPUs (aka CUDA Capable devices) per-node. There is one GPU per-node in the case of Kepler, and two for the Tesla (M2050).

LOGICAL_CPU_COUNT=$(lscpu -p | egrep -v '^#' | wc -l)
PHYSICAL_CPU_COUNT=$(lscpu -p | egrep -v '^#' | sort -u -t, -k 2,4 | wc -l)

The Linux command lscpu is available on Rescale compute nodes. This command gathers CPU architecture information such as the number of CPUs, threads, cores, sockets, NUMA nodes, information about CPU caches, CPU family, model, bogoMIPS, byte order and stepping from sysfs and /proc/cpuinfo, and prints it in a human-readable format. It supports both online and offline CPUs. It can also print out in a parsable format, including how different caches are shared by different CPUs, which can be fed to other programs.

For example, the Rescale Hardware Settings below are for a compute node with the Rescale Luna coretype.

Figure 1: Hardware settings

For this hardware selection, lscpu will report (among other information):

Figure 2: Hardware information

This is equivalent to one physical core and two logical cores. If your code benefits, you may choose to run with two threads.