Run C2Vsim on Cluster

Here I want to show how to run C2Vsim on the Cluster at UCDavis. The cluster OS is Ubuntu and uses SLURM manager for job submission.

To submit a job we have to create shell script, for example the file Run_c2vsim_job.sbatch.

The first part of the script contains options for the SLURM. Below is a small list of options I use to run the jobs

# job name:
#SBATCH --job-name='C2VSIM'
# Number of tasks (cores):
#SBATCH --ntasks=1
#SBATCH --array=1-100
#SBATCH --output=outp%j.log
#SBATCH --error=outp%j.err

The first option  #SBATCH –job-name=’C2VSIM’ defines a name for the job to be submitted.

The second option #SBATCH –ntasks=1 defines the number of cores we need for the job. However IWFM does not support multi-core or multi-thread simulations therefore this option should be 1. However if the simulation requires more memory than the memory allocated by 1 core then we may ask for more cores to increase the memory.

The next option #SBATCH –array=1-100 is used for array jobs. This is used to run repeated simulations with different inputs. This is ideal for Monte Carlo simulations. This option will run the simulation 100 times.

The options #SBATCH –output=outp%j.log and #SBATCH –error=outp%j.err defines the names of the log and error files. For each simulation there would be two files outpXXXXXX.log andoutpXXXXXX.err. The console output is printed on *.log and any errors are printed in *.err file. The XXXXXXX gets its values from the job and array IDs

# Load your modules
# Set up your environment
cd /home/giorgk/C2VSIM/Finegrid

Next we load any module if that’s required. To run IWFM there is no need to load any module and finally we enter into the working directory.

As it was mention in a previous post under linux we need to have all input and output files under the same directory. However this would cause a problem when we try to run the simulation multiple times simultaneously. To do so we need to run each simulation in a separate folder. Doing this manually is quite tedious. To tackle this I use the following workflow: I have created a folder which contains all original C2Vsim input directories and files. Then I copy them to a folder unique for each simulation

echo "------Simulation started $(date)"
cp -r Clean_Input_files $wrksp_folder

The above snippet defines a variable wrksp_folder as temp_id1_id2  where temp can be any user defined string, id1 is the job id given by the SLURM and id2 spans from 1 to number of array jobs. The last command copies the input files from the folder Clean_Input_files to the unique for each simulation folder.

Next we enter into this folder and run the simulation. (Refer to this post where the content of the file is explained)

cd $wrksp_folder

After the simulation is finished we have to copy the results in a safe place and clean the temporary files.

cp -r Results/ ../Results/$res_folder
cd ..
rm -r $wrksp_folder
echo "------Simulation Finished $(date)"

Similar to working folder we define a unique name for the results folder. Next we copy the content of the $wrksp_folder/Results folder somewhere outside of the $wrksp_folder/ (It may seem confusing but in the second line the Results/ and ../Results/ are two totally different directories)

After the results are copied we delete the $wrksp_folder and print the time when the simulation is finished.

To submit this job on the cluster execute

sbatch Run_c2vsim_job.sbatch

However for the farm cluster at UCDavis this is not going to work. Actually you need to run this with the following option

sbatch -p serial Run_c2vsim_job.sbatch



Run C2Vsim on Linux

The California Central Valley Groundwater-Surface Water Simulation Model (C2VSim) is an integrated numerical model that simulates water movement through the linked land surface, groundwater and surface water flow systems in California’s Central Valley.

In this post I’d like to highlight the changes of the input files needed to run C2Vsim on Linux systems.

First we create a similar to *.bat file for linux e.g. an file.

Because I want to switch easily between the debug and release versions of IWFM I’ll have the script switching to debug mode when the user provides a zero input. e.g. ./ 0

At the first lines of the file I define some user input as follows


The first line is the path to the IWFM executables and the second is a temporary name whose purpose will be explained shortly.

The next part of the script identifies which version the user chooses

if [ -z "$mode" ];
if [ $mode -eq 0 ]

Next we enter the Preprocessor folder and run the preprocessing part of the program. The last line moves the output of the program to the simulation folder

cd Preprocessor/
# Run preprocessor
if [ $mode -eq 0 ]
mv CVpreout.bin ../Simulation/CVpreout.bin

Next its time to run the main simulation program. However we need to fix the input and output paths. Unfortunately linux paths are not recognized by IWFM, therefore all input and output files should be located under the simulation folder. This is as easy as to remove the ../Results/ from all input and output filenames inside the file. Once this is taken care, the follow snippet will enter the simulation folder run the model and move the output files from the Simulation folder to Results folder to keep everything neat and clean

cd ../Simulation
# Run Simulation
if [ $mode -eq 0 ]
find $PWD -name '*.bin' -exec mv {} ../Results/ \;
find $PWD -name '*.out' -exec mv {} ../Results/ \;

After the simulation is finished we can run the Budget. Similar to above IWFM does not understands linux paths and we have to put the input files in the same folder as the main input. The following snippet enters the Budget directory and copies all the required inputs from results to the budget folder after appending a prefix defined by the user. This prefix will be handy later to delete those files. However we need to change the filenames of the input budget file. To do so we have to change the ..\Results\ with temp_ or any temp name defined by the user above. Next run the budget and copy the results to the ../Results folder without the prefix temp. Last we delete all the files with temp prefix to keep the folder clean

cd ../Budget
# copy the files needed to run the budget
cp ../Results/CVlandwater.bin ${temp}_CVlandwater.bin
cp ../Results/CVstream.bin ${temp}_CVstream.bin
cp ../Results/CVrootzn.bin ${temp}_CVrootzn.bin
cp ../Results/CVground.bin ${temp}_CVground.bin
cp ../Results/CVsmwshed.bin ${temp}_CVsmwshed.bin
cp ../Results/CVlake.bin ${temp}_CVlake.bin
cp ../Results/CVstreamrch.bin ${temp}_CVstreamrch.bin
cp ../Results/CVdiverdtl.bin ${temp}_CVdiverdtl.bin
#Run budget
if [ $mode -eq 0 ]
cp ${temp}_CVlandwater.BUD ../Results/CVlandwater.BUD
cp ${temp}_CVstream.BUD ../Results/CVstream.BUD
cp ${temp}_CVrootzn.BUD ../Results/CVrootzn.BUD
cp ${temp}_CVground.BUD ../Results/CVground.BUD
cp ${temp}_CVsmwshed.BUD ../Results/CVsmwshed.BUD
cp ${temp}_CVlake.BUD ../Results/CVlake.BUD
cp ${temp}_CVstreamrch.BUD ../Results/CVstreamrch.BUD
cp ${temp}_CVdiverdtl.BUD ../Results/CVdiverdtl.BUD
rm -r ${temp}_*

The last step is to run the Zbudget. For each budget zone we have to repeat the following lines. First we enter the Z-Budget folder and copy the required input from the Results folder to the Z-Budget folder while appending the prefix temp for the same reasons that were explained previously. We have also to modify the namefiles inside the input file as we did previously (replace the ..\Results\ with temp_). After the Zbudget code is executed the result is copied to the ../Results folder without the prefix and finally remove the temporary files.

cd ../Z-Budget
cp ../Results/CVZB.bin ${temp}_CVZB.bin
#Run Z-budget
if [ $mode -eq 0 ]
cp ${temp}_CVZB.BUD ../Results/CVZB.BUD
rm -r ${temp}_*

That concludes the simulation and exit the Z-Budget folder

cd ..


IMPORTANT NOTE: IWFM is case sensitive. CVPrint.dat is different from CVprint.dat