This tutorial shows you how to use a Persistent cluster on the Rescale platform. Please contact us if you're having trouble.
About Persistent Clusters
In the standard Rescale workflow, a new hardware cluster is launched for each job that you submit, and it is terminated automatically once the simulation is completed. While this is advantageous in many ways, one may also want to re-submit jobs to the same cluster.
Persistent Clusters are hardware that remain alive and running even after your job has completed. You can then proceed to submit other jobs onto the same cluster, which will run immediately.
Note: The persistent cluster will NOT shut down automatically. You will continue to be charged per core-hour that the cluster is alive, even if it is idling.
Save Time with Persistent Clusters
Hardware clusters typically take a few minutes to boot up. In the case where you need to submit multiple jobs in series, a significant portion of time is wasted waiting for clusters to spin up and down. With Persistent Clusters, subsequent jobs that use the same configuration can start immediately on the same cluster.
This can be particularly useful if you have a series of jobs that run on similar configurations. Perhaps you need the data from the former run to tweak the subsequent run, and cannot run them concurrently. You can modify your input files or scripts directly on the Rescale UI and submit them to the already running Persistent Cluster. This is also very useful if you are using a new software/feature on Rescale for the first time, or experimenting with setting up a new simulation/model.
For example, if you are setting up a job for the first time and are not sure what commands to use, you can iterate quickly using persistent clusters to figure out the right combination. By also making use of our live tailing feature, you can track the output and progress of the job and restart the jobs at a much faster rate than before, without having to wait for clusters to boot up and shut down.
How to launch a Persistent Cluster
To create a new job, click + New Job on the top-left corner of your screen.
Next, set up your job as you normally would (if unsure, refer here). On the hardware settings page, you will see a checkbox to enable the Reusable cluster feature. Please check this box and launch the job. The cluster that is associated with job is now a persistent cluster and will remain available for use even after this run has completed.
Utilizing an Active Persistent Cluster
Set up a second job like how you normally would. On the hardware settings page, you can select from a list of active Persistent Clusters for your job. Subsequent jobs that are submitted onto a Persistent cluster will be queued and run in the order that they were submitted, as each job completes and the cluster frees up.
Note: You are not able to change the hardware or software configurations on a persistant cluster once it has been launched. Please launch the cluster with all the required software selected beforehand. You can, however, change the input files, post-processing scripts, and the command line on the "Software Settings" page.
Terminating a Persistent Cluster
To terminate a persistent cluster, please navigate to the Clusters tab at the top of the page. Here you can see a list of your past and active clusters. Please name your clusters appropriately during job setup so that you can identify the right cluster easily on this page. Click on the red "Stop" button next to the cluster that you would like to terminate. (See screenshot below)