FAQs

This tutorial shows you how to stop a running CFX Solver from the command line. Similar to pressing the stop button in the ANSYS CFX GUI, inserting a stop file executes the stop and writes a restart file at the end of an outer loop iteration in steady state simulations, or at the end of a timestep in transient analyses.

To complete this task, you will need to use our in-browser SSH. Go to the Status page of your job. Then below the Live Tailing panel, you will see an SSH panel with a thunderbolt symbol. In the SSH panel, you will see a Server IP address which you can click to launch an in-browser SSH terminal. This way, you can interact with the running cluster from your browser.

For more information on using Rescale's in-broswer SSH feature, please refer to the tutorial here.

Stop file option 1 - Using the cfx5stop command

In the terminal window, type the following commands (2 lines):

cd work
cfx5stop -directory <temp-directory>

Suppose a run is called MultiphaseMixer_001 in the work directory. Then, there will be a temporary directory called MultiphaseMixer_001.dir in the work directory. The command will be as follows:

cd work
cfx5stop -directory MultiphaseMixer_001.dir

Stop file option 2 - Writing a stop file

In the terminal window, type the following command:

touch work/<temp-directory>/stp

Suppose a run is called MultiphaseMixer_001 in the work directory. Then, there will be a temporary directory called MultiphaseMixer_001.dir in the work directory. The command will be as follows:

touch work/MultiphaseMixer_001.dir/stp

Restarting a job

Inserting a stop file will generate a .res restart file that can be used to restart the simulation.

In the Results page of the stopped job, you may make the restart file available as an input file (see image below), then clone the job. In the new cloned job, insert the restart file in place of the input file both in the Input Files page, as well as, in the Command in the Software Settings page of the Setup.

Input Files

You can change your license feature on workbench by going to Tools > License Preferences.

License Preferences

In the preferences window, you will automatically see all the licenses that you own. ANSYS Workbench will try to checkout the appropriate license in the order that is specified here, from top to bottom. In order to give priority to a license feature, use the "Move up" and "Move down" feature to bring it to the top. You can also disable certain license features by setting the assigned value to 1 or 0.

In this case where you want to change the license checked out for meshing, please navigate to the PrepPost tab and set up the license feature priority that you require.

License Manager

In the software settings page, you are able to specify the license feature you would like to check out by appending the -p flag to your command.

For example, if you would like to use the feature meba, your command box will look something like this:

ansys181 -dis -b -mpi ibmmpi -np $RESCALE_CORES_PER_SLOT -machines $MACHINES -p meba -i <input-file>

When submitting an ANSYS HFSS job with HPC at Rescale, it is good practice to gather DEBUG files in case the job fails. In the event of a failing analysis, the DEBUG files can then be provided to Rescale Support. Please contact us if you're having trouble.

On the command window of the Software Settings page of your job setup, add the following commands before the Ansys Electronics Desktop command ansysedt that you typically run. An example is shown below:

export DEBUG_LOG_DIR=`pwd`/debug_log
mkdir $DEBUG_LOG_DIR 2>/dev/null
export ANSOFT_DEBUG_LOG=$DEBUG_LOG_DIR/debug
export ANSOFT_DEBUG_LOG_SEPARATE=1
export ANSOFT_DEBUG_LOG_THREAD_ID=1
export ANSOFT_DEBUG_LOG_TIMESTAMP=1
export ANSOFT_DEBUG_MODE=2
export ANSOFT_PASS_DEBUG_ENV_TO_REMOTE_ENGINES=1
ansysedt -auto -distributed -monitor -machinelist file=$HOME/machinefile.hfss -ng -batchsolve test.aedt

In addition to running the analysis, this will create a debug-log directory within the work directory which will contain all the generated logs. It is recommended that you use these commands every time you submit an ANSYS HFSS job in batch.

This tutorial shows you how to insert a check-fluent or exit-fluent file in the job directory of your running cluster in order to stop the iteration and write case/data files.

You will need first set up your SSH to perform this task. If you have not already done so, please follow the guide here. Click on the Server IP address on the SSH panel on the Status page to launch your SSH session, and follow the instructions below.

check-fluent file

A check-fluent file triggers Fluent to write and save case and data files at the end of the current iteration. The simulation will then continue to solve.

To insert a check-fluent file, enter the following commands in your SSH session (2 lines).

cd /tmp
touch check-fluent

For instance, if the journal file is setup to run for 500 iterations and a check-fluent file is inserted during the 28th iteration, then a case and data file will be created with the iteration step appended to the file name. The simulation will continue and complete the 500 iterations. The following files will be available in the Results page of the job.

  • tjunction_plot0028.cas.gz
  • tjunction-plot0028.dat.gz
  • tjunction_plot0500.cas.gz
  • tjunction-plot0500.dat.gz

exit-fluent file

An exit-fluent file, triggers Fluent to write and save case and data files at the end of the current iteration. The simulation will then terminate and exit Fluent.

To insert an exit-fluent file, enter the following commands in your SSH session (2 lines).

cd /tmp
touch exit-fluent

For instance, if the journal file is setup to run for 500 iterations and an exit-fluent file is inserted during the 28th iteration, then a case and data file will be created with the iteration step appended to the file name. The simulation will not continue. The following files will be available in the Results page of the job.

  • tjunction_plot0028.cas.gz
  • tjunction-plot0028.dat.gz

This tutorial shows you how to generate a residual plot during an ANSYS Fluent batch job. We will show two ways how you can accomplish this:

1. Setting it up on the Fluent GUI

You can use this method when you are setting up a job on a Rescale Desktop or your own local workstation.

  • Open the relevant model in Fluent
  • Go to the Solution sub-tree on the main menu. Click on Calculation Activities and then on Execute Commands in the setup tree. This will open a dialog box to Define Commands

Fluent Solution subtree

  • Enable the Active check button next to the command
  • Enter a name for the command under Name
  • Indicate how often you want the residuals plotted by setting the interval under Every and selecting Iteration or Time Step in the drop-down list under When . Time step is a valid option for unsteady flow
  • Under Command type:
/solve/monitors/residual/plot? yes /display/set-window 1 /display/save-picture residual.png
  • Click on OK to save the changes

Fluent Execute Commands

2. Setting it at the journal file

You can use this method when you are setting up a batch job on the Rescale platform.

  • Enter the following into the journal file after initialization and before solving
/solve/execute-commands/add-edit command-1 500 "iteration" "/solve/monitors/residual/plot? yes /display/set-window 1 /display/save-picture residual.png"

Note:

In the command line above, command-1 500 "iteration" allows to print residuals every 500 iterations. 500 iterations is used here just as an example and may not be suitable for all jobs. Determine the frequency appropriate for the length of your job and modify command-1 500 "iteration" as needed. Rewriting the residuals too often may cause the job to slow down.

A sample journal file with residual plot setup is shown below. Here, the run time is 7500 iterations and residuals are plotted every 500 iterations.

/file/set-batch-options no yes yes no
/file/rc tjunction_plot.cas
/solve/init/init/
/solve/execute-commands/add-edit command-1 500 "iteration" "/solve/monitors/residual/plot? yes /display/set-window 1 /display/save-picture residual.png" 
/solve/iterate 7500
/file/wcd tjunction_plot%i.cas.gz
/exit yes

Live Tailing and Results

You can view the residuals plot during the batch run using the live tailing feature. Once the job has started running, the Live Tailing panel is displayed. The residuals.png file should be created after the first few iterations. Search for `.png' file. Click on the file to display the plot.

Live Tailing

Once the job is completed, the residuals plot file is populated on the Results page along with the other job files. Click on View icon to view the file. You can also download the residuals plot file by clicking on Download icon or download all job files by clicking on the Download Job icon.

Fluent Results Files

ANSYS Electronics Desktop supports two types of HPC licenses: Pack and Pool licenses. When you submit a batch job through the ScaleX platform such as HFSS or Maxwell, the default command is set up to use the HPC Pack license. In order to use your HPC pool license, you need to add the following flag to your command:

For HFSS:

-batchoptions "'HFSS/HPCLicenseType'='Pool'"

For Maxwell:

-batchoptions "'Maxwell 3D/HPCLicenseType'='Pool'"

A sample HFSS execution commmand containing the HPC pool license flag is shown below:

export TASKS_PER_NODE=-1
for host in `cat $HOME/machinefile`; do echo "$host:$TASKS_PER_NODE:$RESCALE_CORES_PER_NODE:90%" >> $HOME/work/machinefile.hfss; done
ansysedt -auto -distributed -monitor -machinelist file=$HOME/work/machinefile.hfss -batchoptions "'HFSS/HPCLicenseType'='Pool'" -ng -batchsolve my-hfss-input-file.aedt

Similarly, a sample Maxwell execution command containing the HPC pool license flag is shown below:

export TASKS_PER_NODE=8
for host in `cat $HOME/machinefile`; do echo "$host:$TASKS_PER_NODE:$RESCALE_CORES_PER_NODE:90%" >> $HOME/work/machinefile.maxwell; done
ansysedt -distributed -monitor -machinelist file=$HOME/work/machinefile.maxwell -ng -batchsolve my-maxwell-input-file.aedt

ANSYS Electronics has the option to specify the tasks automatically or manually for a distributed batch job. Please refer to the ANSYS Electromagnetics HPC Administrator's Guide for more information about automatic and manual mode.

Automatic Mode

In this mode ANSYS Electronics software determines how the tasks should be distributed to the available resources. In the command line, we use an environment variable, TASK_PER_NODE. When this variable is set to -1, the tasks will be automatically distributed among the cores. The -auto flag is also used to specify automatic mode. The command line will look as follows for automatic mode -

For HFSS :

export TASKS_PER_NODE=-1
for host in `cat $HOME/machinefile`; do ssh ${host} "ansoftrsmservice start; RegisterEnginesWithRSM.pl add;"; echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.hfss; done 
ansysedt -auto -distributed -monitor -machinelist file=$HOME/work/ machinefile.hfss -ng -batchsolve ansys-hfss-electronics-example.aedtz

For Maxwell :

export TASKS_PER_NODE=-1
for host in `cat $HOME/machinefile`; do echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.maxwell; done
ansysedt -auto -distributed -monitor -machinelist file=$HOME/work/machinefile.maxwell -ng -batchsolve rotational-actuator.aedtz

Manual Mode

In this mode, users can specify the total number of cores and number of tasks per node (And hence, number of cores per task). In the command line, we can set the the environment variable, TASKS_PER_NODE to ' 2 ' , ' 4 ' , ' 8 ', ' 16 ' etc. Additionally, we need to delete the -auto flag.

For example, if we're using 8 cores for a job and we're setting tasks per node as 2, then each task will require 4 cores and this will result in a maximum of two tasks per host.

For the above example, the commands line on the Rescale platform will be as follows :

For HFSS :

export TASKS_PER_NODE=2
for host in `cat $HOME/machinefile`; do ssh ${host} "ansoftrsmservice start; RegisterEnginesWithRSM.pl add;"; echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.hfss; done 
ansysedt -distributed -monitor -machinelist file=$HOME/work/ machinefile.hfss -ng -batchsolve ansys-hfss-electronics-example.aedtz

For Maxwell :

export TASKS_PER_NODE=2
for host in `cat $HOME/machinefile`; do echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.maxwell; done
ansysedt -distributed -monitor -machinelist file=$HOME/work/machinefile.maxwell -ng -batchsolve rotational-actuator.aedtz

The default command line for ANSYS Electronics on Rescale platform has the placeholder . You can specfify one of the design option types in the placeholder by including the expression within quotes :

  • "[designName]" - batch solve all setups for design with the name given under the project.

  • "[designName]:Nominal" - batch solve all nominal setups for design with the name given under the project.

  • "[designName]:Optimetrics" - batch solve all Optimetrics setups for design with the name given under the project.

  • "[designName]:Nominal:[setupname]" - batch solve the specified nominal setup for design with the name given under the project. The setupname is case insensitive.

  • "[designName]:Optimetrics:[setupname] - batch solve the specified Optimetrics setup for design with the name given under the project. The setupname is case insensitive.

For HFSS :

In the example problem given in the ANSYS HFSS batch example, if we want to solve the Nominal part of second design named "pcie_gen3_fab3_cutout", then the command line will look as follows -

export TASKS_PER_NODE=-1
for host in `cat $HOME/machinefile`; do ssh ${host} "ansoftrsmservice start; RegisterEnginesWithRSM.pl add;"; echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.hfss; done 
ansysedt -auto -distributed -monitor -machinelist file=$HOME/work/ machinefile.hfss -ng -batchsolve "pcie_gen3_fab3_cutout:Nominal" ansys-hfss-electronics-example.aedtz

For Maxwell :

To solve only the nominal part given in the ANSYS Maxwell batch example, the command line will look like this -

export TASKS_PER_NODE=2
for host in `cat $HOME/machinefile`; do echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.maxwell; done
ansysedt -distributed -monitor -machinelist file=$HOME/work/machinefile.maxwell -ng -batchsolve "Rotational_actuator:Nominal" rotational-actuator.aedtz

If the user is running HFSS/Maxwell on Azure, the IB environment variables need to be set before the ansys command lines and additional batch options (-batchoption ” ‘<SOFTWARE>/MPIVendor’=‘Intel’ ”) need to be specified.

For HFSS :

export I_MPI_DAPL_PROVIDER=ofa-v2-ib0
export I_MPI_DYNAMIC_CONNECTION=0
export I_MPI_DAPL_UD=0
export I_MPI_DAPL_TRANSLATION_CACHE=0
export I_MPI_FABRICS=shm:dapl
export TASKS_PER_NODE=-1
for host in `cat $HOME/machinefile`; do ssh ${host} "ansoftrsmservice start; RegisterEnginesWithRSM.pl add;"; echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.hfss; done 
ansysedt -auto -distributed -monitor -machinelist file=$HOME/work/ machinefile.hfss -batchoption ” ‘HFSS/MPIVendor’=‘Intel’ ” -ng -batchsolve ansys-hfss-electronics-example.aedtz

For Maxwell :

export I_MPI_DAPL_PROVIDER=ofa-v2-ib0
export I_MPI_DYNAMIC_CONNECTION=0
export I_MPI_DAPL_UD=0
export I_MPI_DAPL_TRANSLATION_CACHE=0
export I_MPI_FABRICS=shm:dapl
export TASKS_PER_NODE=2
for host in `cat $HOME/machinefile`; do echo ${host}:${TASKS_PER_NODE}:${RESCALE_CORES_PER_NODE}:90% >> $HOME/work/machinefile.maxwell; done
ansysedt -distributed -monitor -machinelist file=$HOME/work/machinefile.maxwell -batchoption ” ‘Maxwell 3D/MPIVendor’=‘Intel’ ” -ng -batchsolve rotational-actuator.aedtz