###################### A guide to using SETTE ###################### .. contents:: :local: Overview ======== ``SETTE`` is a suite of bash scripts that automates the building, running and basic validation and verification of a broad spectrum of NEMO reference and test configurations. Because compilation and batch running environments differ wildly, automation is only achieved after some effort by the user for each new test platform. However, examples are provided for all the major systems used by the NEMO System team and many new platforms can be incorporated simply by adapting these templates. When configured correctly, a single command will: * Compile multiple reference and test configurations * Run restartability and reproducibility tests with each configuration * Run additional conformance checks with any AGRIF-based configurations * Archive sufficient output measures for meaningful comparsion between future tests at different revisions * On completion, run a secondary script to table the successes or failures of each test Many namelist-controlled options can be varied using command line arguments to the main sette script and test results can be compared across different innvocations. Thus, by chaining sette innvocations with different options, more complex and comprehensive testing can be achieved. Installation ============ ``SETTE`` is provided within the main NEMO git repository and will be found in the subdirectory ``sette`` below the top-level of a checked out (cloned) copy (at the same level as ``src/`` or ``cfgs/``). .. _Initial setup: Initial setup ============= Assuming, for now, that you are intending to run ``SETTE`` on one of the platforms already supported then there are only a few settings required to setup for each individual user. These settings are all to be found in the ``sette/param.default`` file: .. code-block:: bash NEMO_VALIDATION_REF=/path/to/reference/sette/results NEMO_REV_REF=0000 COMPILER=${SETTE_COMPILER:-XXXXXXXX} BATCH_CMD=${SETTE_BATCH_CMD:-llsubmit} BATCH_STAT=${SETTE_BATCH_STAT:-llq} FORCING_DIR=${SETTE_FORCING_DIR:-$WORKDIR/FORCING} NEMO_VALIDATION_DIR=${SETTE_NEMO_VALIDATION_DIR:-$MAIN_DIR}/NEMO_VALIDATION JOB_PREFIX_NOMPMD=${SETTE_JOB_PREFIX_NOMPMD:-batch} JOB_PREFIX_MPMD=${SETTE_JOB_PREFIX_MPMD:-batch-mpmd} and each can be set either in a ``param.cfg`` file (created by copying ``param.default`` to ``param.cfg`` and editing) or through the corresponding environment variable. For example, changing the contents of a ``param.cfg`` to include: .. code-block:: bash NEMO_VALIDATION_REF=/work/n01/n01/acc/NEMO/2022/4.2.0/sette/NEMO_VALIDATION NEMO_REV_REF=14550 COMPILER=X86_ARCHER2-Cray BATCH_CMD=sbatch BATCH_STAT=squeue FORCING_DIR=/work/n01/n01/acc/FORCING/SETTE_inputs/r4.2.0 NEMO_VALIDATION_DIR=/work/n01/n01/acc/NEMO/2022/4.2.0/sette/NEMO_VALIDATION JOB_PREFIX_NOMPMD=batch JOB_PREFIX_MPMD=batch or settings: .. code-block:: bash export SETTE_COMPILER=X86_ARCHER2-Cray export SETTE_BATCH_CMD=sbatch export SETTE_BATCH_STAT=squeue export SETTE_FORCING_DIR=/work/n01/n01/acc/FORCING/SETTE_inputs/r4.2.0 export SETTE_NEMO_VALIDATION_DIR=/work/n01/n01/acc/NEMO/2021/4.2.0/sette/NEMO_VALIDATION export SETTE_JOB_PREFIX_NOMPMD=batch export SETTE_JOB_PREFIX_MPMD=batch in your runtime environment will achieve the same result. The requirement to create a ``param.cfg`` from ``param.default`` for each installation protects against developers accidentally returning a local ``param.cfg`` to the main repository. The ``param.default`` file should only be altered by ``SETTE`` developers. .. note:: Apart from ``NEMO_VALIDATION_REF`` and ``NEMO_REV_REF`` which do not have an equivalent environment variable - TO BE FIXED The purposes of these settings should be clear: * ``NEMO_VALIDATION_REF`` points to a ``SETTE``-generated directory of previous archived results to be used optionally as a reference set. * ``NEMO_REV_REF`` defines the reference revision number (Q: how to adapt this for git?) * ``COMPILER`` names the architecture file to be used to compile NEMO * ``BATCH_CMD`` names the command used to submit jobs to the batch system * ``BATCH_STAT`` names the command used to query the batch system and return a list of queued and running jobs * ``FORCING_DIR`` points to a directory containing the input files required by the reference configurations. Details of how to obtain the files with which to populate this directory are provided in the `Obtaining configuration input files`_ section. Note that this directory must be in a part of the filesystem that is visible to the back-end compute nodes. * ``NEMO_VALIDATION_DIR`` points to a directory under which ``SETTE`` will archive its results. * ``JOB_PREFIX_NOMPMD`` A prefix for the template batch file if running in SPMD mode (e,g. with attached XIOS servers) * ``JOB_PREFIX_MPMD`` An alternative prefix for the template batch file if running in MPMD mode (i.e. with detached XIOS servers). This isn't necessarily different to the SPMD setting since some of the templates provided are written to handle both modes. See the `Template batch files`_ section for details. .. _`Obtaining configuration input files`: Obtaining configuration input files =================================== Many of the reference configurations require domain files, initial conditions and surface forcing fields. The exceptions are the GYRE_PISCES configurations and the simpler, test-cases. It is possible to limit ``SETTE`` to a subset of tests to avoid the need for downloading data files but far less of the code will be covered by the tests in this case. All the required files are available from the `SETTE inputs site`_ and there is an option to select a ``LITE`` set of inputs which reduces the volume by compromising on the duration of forcing data sets and the number of significant digits used. NEMO developers will test releases with the standard set of inputs but the ``LITE`` set is suitable for quick deployment and tests will still be capable of highlighting issues with restartability or reproducibility. Just be aware that answers will vary between tests using the standard set and the ``LITE`` set and ensure consistent inputs have been used when comparing different sets of test results. The entire set of inputs can be downloaded using the ``./sette_fetch_inputs.sh`` script which uses ``wget`` to retrieve the files and populate the ``FORCING_DIR`` directory. This script accepts a ``-l`` argument which forces the retrieval of the ``LITE`` alternatives. I.e.: .. code-block:: bash sette_fetch_inputs.sh -h sette_fetch_inputs.sh : Fetch 4.2.0 input files from remote store -l Fetch the alternative, 4.2.0_LITE input files from remote store If you are using the ``LITE`` versions then you will, additionally, have to execute the ``sette_use_LITE.sh`` script which will edit the internal ``cfg`` file for each configuration to register the alternative names. This only needs to be done once if you intend to continue using the ``LITE`` set. ``sette_use_LITE.sh`` accepts the ``-r`` argument to reverse these changes if toggling between using the standard and ``LITE`` inputs. .. _`SETTE inputs site`: https://gws-access.jasmin.ac.uk/public/nemo/sette_inputs/ .. _Template batch files: Template batch files ==================== If you have previously compiled NEMO successfully on your test platform then you can have confidence that providing the same environment and arch file to ``SETTE`` will allow SETTE to compile successfully (barring any compile-time bugs in previously untested code). However, ``SETTE`` also needs to configure and run a series of batch jobs with varying resource requirements. To do this, you must provide ``SETTE`` with a means to generate valid job submission scripts. There are essentially two ways of doing this: * Provide a template batch file with known strings that ``SETTE`` can replace with settings based on the needs of each job. * Provide an external script that can accept those settings and generate a new batch file. If you are working on a test platform already supported by ``SETTE``, a solution will already be in place and you can skip to the `Component scripts`_ section. If you are commissioning a new platform then you will need to provide either a template batch file or an external generating script. Batch file templates are located in the ``sette/BATCH_TEMPLATE`` subdirectory. They are named with the ``COMPILER`` setting prefixed by the ``JOB_PREFIX_NOMPMD`` (or ``JOB_PREFIX_NOMPMD``) setting and separated by a hyphen. To use the first method, the batch template should be a version of a working submission script with the following strings in place of their corresponding numerical values: .. code-block:: bash NODES The total number of nodes required TOTAL_NPROCS The total number of cores required NPROCS The number of ocean cores NXIOPROCS The number of XIOS cores MPI_FLAG A logical to declare if MPI is being used DEF_EXE_DIR The test execution directory Paths and names that have to be passed through to the post-run tidy-up script: DEF_SETTE_DIR DEF_INPUT_DIR DEF_CONFIG_DIR DEF_TOOLS_DIR DEF_NEMO_VALIDATION DEF_NEW_CONF DEF_CMP_NAM DEF_TEST_NAME Not all of these need to used and if your particular system needs additional information then this can be added as a case statement (dependent on ``COMPILER``) in the ``prepare_job.sh`` script An example template file is given in the sette_batch_template file: .. code-block:: bash cat BATCH_TEMPLATE/sette_batch_template #!/bin/bash #! # @ job_name = MPI_config # standard output file # @ output = $(job_name).$(jobid) # standard error file # @ error = $(job_name).$(jobid) # job type # @ job_type = parallel # Number of procs # @ total_tasks = NPROCS # time # @ wall_clock_limit = 0:30:00 # @ queue # # Test specific settings. Do not hand edit these lines; the prepare_job.sh script will set these # (via sed operating on this template job file). # OCEANCORES=NPROCS export SETTE_DIR=DEF_SETTE_DIR ############################################################### # # set up mpp computing environment # # Local settings for machine IBM Power6 (VARGAS at IDRIS France) # export MPIRUN="mpiexec -n $OCEANCORES" # # load sette functions (only post_test_tidyup needed) # . ${SETTE_DIR}/all_functions.sh # Do not remove or change the following comment line # BODY # # These variables are needed by post_test_tidyup function in all_functions.sh # export EXE_DIR=DEF_EXE_DIR export INPUT_DIR=DEF_INPUT_DIR export CONFIG_DIR=DEF_CONFIG_DIR export TOOLS_DIR=DEF_TOOLS_DIR export NEMO_VALIDATION_DIR=DEF_NEMO_VALIDATION export NEW_CONF=DEF_NEW_CONF export CMP_NAM=DEF_CMP_NAM export TEST_NAME=DEF_TEST_NAME # # end of set up ############################################################### # # change to the working directory # cd ${EXE_DIR} echo Running on host `hostname` echo Time is `date` echo Directory is `pwd` # # Run the parallel MPI executable # if [ MPI_FLAG == "yes" ]; then echo "Running time ${MPIRUN} ./nemo" time ${MPIRUN} ./nemo else echo "Running time./nemo" time ./nemo fi # post_test_tidyup # END_BODY # Do not remove or change the previous comment line exit But commamds and environments are most likely different on every test platform so it may require some effort to produce an equivalent template for new platforms. For cases where the calculation and declaration of resources is more complex (for example, in hetrogeneous computing environments requiring job-pack submissions), in may be easier to provide an external script to generate the job script. An example is included in the case of X86_ARCHER2-Cray where the batch template is simply a placeholder: .. code-block:: bash #!/bin/bash # # A batch script will be generated using: # /work/n01/shared/acc/mkslurm_settejob -S $NXIO_PROC -s 8 -m 4 -C $NB_PROC -g 2 -a n01-CLASS -j sette_job -t 20:00 > ${SETTE_DIR}/job_batch_template # by prepare_job.sh # and the suggested script is executed by prepare_job.sh instead of editing the template: .. code-block:: bash case ${COMPILER} in X64_MOBILIS*) . . ;; X86_ARCHER2*) MK_TEMPLATE=$( /work/n01/shared/acc/mkslurm_settejob_4.2 -S $NXIO_PROC -s 8 -m 4 -C $NB_PROC -g 2 -a n01-CLASS -j sette_job -t 20:00 > ${SETTE_DIR}/job_batch_template ) ;; Any such solutions should be fed back to the system team for incorporation into future releases. .. _Component scripts: Component scripts ================= ``SETTE`` consists of a suite of scripts and settings files. A complete list is given here but basic use of ``SETTE`` only requires familiarisation with the first two listed: + **User scripts and settings** - ``param.cfg`` - ``sette.sh`` * ``sette_reference-configurations.sh`` * ``sette_test-cases.sh`` - ``sette_rpt.sh`` - ``sette_eval.sh`` - ``sette_fetch_inputs.sh`` - ``sette_list_avail_cfg.sh`` - ``sette_list_avail_rev.sh`` - ``sette_use_LITE.sh`` + **Internal scripts and settings** - ``all_functions.sh`` - ``fcm_job.sh`` - ``prepare_exe_dir.sh`` - ``prepare_job.sh`` - ``input_.cfg`` .. _Usage of main scripts: Usage of main scripts ===================== The purpose and contents of ``param.cfg`` were explained in the `Initial setup`_ section. ``sette.sh`` is the main utility script that, when executed without any arguments, will compile, configure and submit a pre-set series of tests. After all the tests have completed, a basic report is presented to the user which lists the various successes or failures. .. code-block:: bash ./sette.sh Current code is : NEMO/trunk @ r15544 ( last change @ r15541 ) SETTE validation report generated for : NEMO/trunk @ r15541 (last changed revision) on X86_ARCHER2-Cray arch file !!---------------1st pass------------------!! !----restart----! GYRE_PISCES run.stat restartability passed : 15541 GYRE_PISCES tracer.stat restartability passed : 15541 ORCA2_ICE_PISCES run.stat restartability passed : 15541 ORCA2_ICE_PISCES tracer.stat restartability passed : 15541 ORCA2_OFF_PISCES tracer.stat restartability passed : 15541 AMM12 run.stat restartability passed : 15541 ORCA2_SAS_ICE run.stat restartability passed : 15541 AGRIF_DEMO run.stat restartability passed : 15541 AGRIF_DEMO tracer.stat restartability passed : 15541 WED025 run.stat restartability passed : 15541 ISOMIP+ run.stat restartability passed : 15541 OVERFLOW run.stat restartability passed : 15541 LOCK_EXCHANGE run.stat restartability passed : 15541 VORTEX run.stat restartability passed : 15541 ICE_AGRIF run.stat restartability passed : 15541 SWG run.stat restartability passed : 15541 !----repro----! GYRE_PISCES run.stat reproducibility passed : 15541 GYRE_PISCES tracer.stat reproducibility passed : 15541 ORCA2_ICE_PISCES run.stat reproducibility passed : 15541 ORCA2_ICE_PISCES tracer.stat reproducibility passed : 15541 ORCA2_OFF_PISCES tracer.stat reproducibility passed : 15541 AMM12 run.stat reproducibility passed : 15541 ORCA2_SAS_ICE run.stat reproducibility passed : 15541 ORCA2_ICE_OBS run.stat reproducibility passed : 15541 AGRIF_DEMO run.stat reproducibility passed : 15541 AGRIF_DEMO tracer.stat reproducibility passed : 15541 WED025 run.stat reproducibility passed : 15541 ISOMIP+ run.stat reproducibility passed : 15541 VORTEX run.stat reproducibility passed : 15541 ICE_AGRIF run.stat reproducibility passed : 15541 SWG run.stat reproducibility passed : 15541 !----agrif check----! ORCA2 AGRIF vs ORCA2 NOAGRIF run.stat unchanged - passed : 15541 15541 !----result comparison check----! check result differences between : VALID directory : /work/n01/n01/acc/NEMO/4.2.0/sette/NEMO_VALIDATION/MAIN at rev 15541 and REFERENCE directory : /work/n01/n01/acc/NEMO/4.2.0/sette/NEMO_VALIDATION/MAIN at rev 15150 GYRE_PISCES run.stat files are identical GYRE_PISCES tracer.stat files are identical ORCA2_ICE_PISCES run.stat files are DIFFERENT (results are different after 1 time steps) ORCA2_ICE_PISCES tracer.stat files are DIFFERENT (results are different after 1 time steps) ORCA2_OFF_PISCES tracer.stat files are DIFFERENT (results are different after 1 time steps) AMM12 run.stat files are DIFFERENT (results are different after 1 time steps) ORCA2_SAS_ICE run.stat files are DIFFERENT (results are different after 3 time steps) AGRIF_DEMO run.stat files are DIFFERENT (results are different after 1 time steps) AGRIF_DEMO tracer.stat files are DIFFERENT (results are different after 1 time steps) WED025 run.stat files are DIFFERENT (results are different after 1 time steps) ISOMIP+ run.stat files are DIFFERENT (results are different after 1 time steps) VORTEX run.stat files are DIFFERENT (results are different after 1 time steps) ICE_AGRIF run.stat files are DIFFERENT (results are different after 2 time steps) OVERFLOW run.stat files are identical LOCK_EXCHANGE run.stat files are identical SWG run.stat files are identical Report timing differences between REFERENCE and VALID (if available) : GYRE_PISCES ref. time: 22.805 cur. time: 40.126 diff.: 17.321 ORCA2_ICE_PISCES ref. time: 133.614 cur. time: 63.484 diff.: -70.13 ORCA2_OFF_PISCES ref. time: 172.469 cur. time: 471.569 diff.: 299.1 AMM12 ref. time: 139.546 cur. time: 222.412 diff.: 82.866 WED025 ref. time: 462.350 cur. time: 913.722 diff.: 451.372 ISOMIP+ ref. time: 33.319 cur. time: 69.091 diff.: 35.772 OVERFLOW ref. time: 16.864 cur. time: 35.474 diff.: 18.61 LOCK_EXCHANGE ref. time: 11.912 cur. time: 13.802 diff.: 1.89 The report shows the result of restartability and reproducibility tests on the whole range of test configurations. Passing these tests is a necessary and mandatory requirement for any official release of NEMO. Note these tests are not sufficient to guarantee restartability and reproducibility in all user-defined configrations and anyone running configurations, which are not close variants of the reference or test configurations, should conduct their own tests. This report ends by comparing the latest results against a reference set (as defined in ``param.cfg``). In this case the comparison is between revisions that were known to introduce numerical differences and between runs with different levels of compiler optimisation. This is confined by the comparsion but the report is most useful when numerical results are not expected to change between revisions and when changes are expected to provide a performance benefit. It is not shown here but, on many terminals, test failures or performance drops are presented in red to highlight areas of concern. The set of tests executed by default are set in ``param.cfg`` in the ``TEST_CONFIGS`` environment variable: .. code-block:: bash grep TEST_CONFIGS= param.cfg export TEST_CONFIGS=(${SETTE_TEST_CONFIGS[@]:-"ORCA2_ICE_PISCES ORCA2_OFF_PISCES AMM12 AGRIF WED025 GYRE_PISCES SAS ORCA2_ICE_OBS SWG ICE_AGRIF OVERFLOW LOCK_EXCHANGE VORTEX ISOMIP+"}) Note this set can be overridden by externally setting the ``SETTE_TEST_CONFIGS`` environment variable but individual or sub-sets of tests can also be selected by arguments to the ``-n`` option to sette.sh. This is more explicit and the recommended method since release 4.2. Other options to ``sette.sh`` can be listed using the ``-h`` argument: .. code-block:: bash ./sette.sh -h sette.sh with no arguments (in this case all configuration will be tested with default options) -T to set ln_timing false for all non-AGRIF configurations (default: true) -t set ln_tile false in all tests that support it (default: true) -e set nn_hls=1 (default: nn_hls=2) -i set ln_icebergs false (default: true) -C set nn_comm=1 (default: nn_comm=2 ==> use MPI3 collective comms) -N set ln_nnogather false for ORCA2 configurations (default: true) -q to remove the key_qco key (default: added) -X to remove the key_xios key (default: added) -F to remove the key_loop_fusion key (default: added) -Q to remove the key_RK3 key (currently a null-op since key_RK3 is not used) -A to run tests in attached (SPMD) mode (default: MPMD with key_xios) -n "CFG1_to_test CFG2_to_test ..." to test some specific configurations -x "TEST_type TEST_type ..." to specify particular type(s) of test(s) to run after compilation TEST_type choices are: RESTART REPRO CORRUPT PHYOPTS - anything else will COMPILE only -v "subdir" optional validation record subdirectory to be created below NEMO_VALIDATION_DIR -g "group_suffix" single character suffix to be appended to the standard _ST suffix used for SETTE-built configurations (needed if sette.sh invocations may overlap) -r to execute without waiting to run sette_rpt.sh at the end (useful for chaining sette.sh invocations) -d to perform a dryrun to simply report what settings will be used -c to clean each configuration -s to synchronise the sette MY_SRC and EXP00 with the reference MY_SRC and EXPREF -u to run sette.sh without any user interaction. This means no checks on creating directories etc. i.e. no safety net! The first 11 options are switches to toggle commonly used namelist options or compile time keys. The default setting is to have the option set ``.true.`` or the corresponding key added. Thus, for example, running: .. code-block:: bash ./sette.sh -i -F will run the full suite without icebergs (i.e. ``ln_icebergs=.false.``) and without ``key_loop_fusion`` added at compile time. Some of these options (``-i``, for example) will only affect those configurations that activate the related code option by default. The ``-n`` option allows a sub-set of tests to be named on the command line. Testing will be restricted to the named tests; multiple tests can be listed as a quoted, space-separated string. For example: .. code-block:: bash ./sette.sh and ./sette.sh -n "ORCA2_ICE_PISCES ORCA2_OFF_PISCES AMM12 AGRIF WED025 GYRE_PISCES SAS ORCA2_ICE_OBS SWG ICE_AGRIF OVERFLOW LOCK_EXCHANGE VORTEX ISOMIP+" are equivalent. As a point of interest, it is good practise to list tests in decreasing order of their expected execution time. This will enable compilation time to overlap run-time as much as possible and should minimise time to completion The default operation is to perform all ``RESTART``, ``REPRO`` and agrif-processed, ``CORRUPT`` checks. The checks performed can be limited to a sub-set of these by supplying arguments to the ``-x`` option. The combination of ``-n`` and ``-x`` is particularly useful when working to solve a specific issue with a single configuration. There is also a ``PHYOPTS`` check which is not currently used but has been implemented, as a demonstration, to run the ``OVERFLOW`` and ``LOCK_EXCHANGE`` test cases with a selection of different physical schemes. Any other string supplied as an argument to the ``-x`` option will force a compilation only. This is useful for quickly checking for compile-time errors. Although any non-recognised string will trigger this, it is good practise to be explicit, i.e.: .. code-block:: bash ./sette.sh -x COMPILE The ``-s`` option forces a synchronisation of ``MY_SRC`` contents and input files from the EXREF directory of the reference configuration on which the test is based. This is useful if you know these contents have been changed but do not wish to enforce a complete rebuild. The ``-c`` option forces a clean rebuild from scratch of the test configuration. The ``*_ST?`` directory will be deleted and recreated before a complete compilation and run is performed. Finally, the ``-u`` option disables any interaction with the user. By default, ``sette.sh`` will request confirmation from the user for actions such as: creating the base ``NEMO_VALIDATION_DIR`` or disabling options when incompatible choices are selected. This interaction can be problematic with continuous integration systems and the ``-u`` option should always be used in these applications. It is the responsibility of the user to ensure that the correct information is provided to sette.sh in these cases. .. _The test results archive: The test results archive ======================== This latest version of ``SETTE`` (released with NEMO v4.2) changes the organisation of the records kept under the ``NEMO_VALIDATION_DIR``. This is partly to accommodate the fact that the new command-line options provide so much flexibility for running a series of tests on any one revision of the code with different options. To facilitate such testing, two new command-line options have been introduced: ``-v `` and ``-g ``. ``-v`` names a sub-directory to create (or re-use) beneath ``NEMO_VALIDATION_DIR`` as the root of the records tree. If the ``-v`` option is not used then root of the directory tree will be the branch name as returned by the: .. code-block:: bash git branch --show-current command (or ``MAIN`` if this command fails for any reason). ``-g`` names a single character suffix that will be appended to the traditional ``_ST`` suffix that is added to the configurations built for testing. I.e.: .. code-block:: bash ./sette.sh -e -v HALO1 -g 0 will compile and run the full suite with ``nn_hls=1``. The configurations will be constructed with names such as: ``GYRE_PISCES_ST0`` and the directory structure eventually populated with the test records would be similar to: .. code-block:: bash ./NEMO_VALIDATION |-HALO1 |---X86_ARCHER2-Cray |-----21334_a2c5986+ |-------AGRIF_DEMO |---------LONG |---------ORCA2 |---------REPRO_2_8 |---------REPRO_4_4 |---------SHORT |-------AGRIF_DEMO_NOAGRIF |---------ORCA2 |-------AMM12 |---------LONG |---------REPRO_4_8 |---------REPRO_8_4 |---------SHORT |-------GYRE_PISCES |---------LONG |---------REPRO_2_4 |---------REPRO_4_2 |---------SHORT . . . Use of the ``-g`` option isn't always necessary. In this case, for example, ``-e`` only triggers a namelist changes so there is no difference in the compiled code between this set and the default set (which will use names such as ``GYRE_PISCES_ST`` and would have records stored under ``branch-name``). Thus, running tests sequentially such as: .. code-block:: bash ./sette.sh ./sette.sh -e -v HALO1 will reuse the same run-time directories and only require one set of compilations. However, it will not be possible to diagnose any issues with the first set after the second has run. The use of ``-g`` is recommended when running multiple tests with different compilation keys since future tests with updated code may only need to recompile changed modules and dependencies. Note also that the move from ``subversion`` to ``git`` forces a change in the revision tag used to identify the code base being tested. Whereas, with subversion, the revision number was a integer that increased monotonically in the time-order of commits, ``git`` identifies its commits with long, hexadecimal hash strings that are not necessarily correctly time-ordered when listed alphanumerically. The: .. code-block:: bash git rev-list --abbrev-commit -1 origin command can be used to obtain a abbreviated commit hash that still provides a unique identifier but extra steps are required to provide a revision_tag that retains some indication of time-order. The current solution is to prepend the abbreviated hash string with a representation of the date on which the commit was made. This information can be obtained from the ``git log`` response as follows: .. code-block:: bash git log -1 | grep Date | sed -e 's/.*Date: *//' -e's/ +.*$//' Mon Dec 6 15:24:36 2021 Condensing this string into something usable requires use of the unix ``date`` command which can vary in different flavours of the OS. Two examples are currently supported, one for MacOSX and one, more general, POSIX variety. More can be added in ``param.cfg`` as required. Each supported style is tested in param.cfg to determine which form to use: .. code-block:: bash # command for converting date (from git log -1) into 2-digit year + yearday # date -j -f "%a %b %d %H:%M:%S %Y" "Tue Nov 30 17:10:53 2021" +"%y%j" >& /dev/null if [ $? == 0 ] ; then DATE_CONV='date -j -f "%a %b %d %H:%M:%S %Y" ' ;fi # date --date="Tue Nov 30 17:10:53 2021" +"%y%j" >& /dev/null if [ $? == 0 ] ; then DATE_CONV='date --date=' ;fi In both cases, the output format is a 2-digit year + a 3-digit year-day resulting in a 5-digit string to prepend to the short hash. There is still scope for slight mis-ordering between commits committed on the same day but this compromise avoids over-long revision tags. In the example directory listing above the revision tag is shown as: ``21334_a2c5986+`` which displays a typical 5-digit date and 7-digit short hash separated by an underscore. In this case a ``+`` has been appended because ``sette`` has detected local changes to the base commit. The output of: .. code-block:: bash git status --short -uno is used to check for local modifications when making this decision. Super-sized SETTE ================= The idea of chaining a series of tests to test various options leads to the possibility of a ``super_sette.sh`` script. For example, this content: .. code-block:: bash #!/bin/bash # set -vx # Simple script to robustly run a full suite of SETTE tests # ######################################## # Start of main script ######################################## FULLSET=( ORCA2_ICE_PISCES ORCA2_OFF_PISCES AMM12 AGRIF WED025 GYRE_PISCES SAS ORCA2_ICE_OBS SWG ICE_AGRIF OVERFLOW LOCK_EXCHANGE VORTEX ISOMIP+ ) # GROUP_SETS=( "-g 0 -r -v MAIN" "-e -F -t -g 1 -v HALO1 -r" "-q -g 2 -v NO_QCO -r" "-i -e -F -t -g 3 -v NO_ICB1 -r" "-i -g 4 -v NO_ICB2 -r" "-C -g 5 -v NO_COLL -r" ) # # These groups sets correspond to the following test regimes: # # A. Three complete sets with various combinations of options: # printf "%-93s %s\n" "Full tests - MAIN with default options (using *_ST0 config dirs) : " "${GROUP_SETS[0]}" printf "%-93s %s\n" "Full tests - HALO1 with nn_hls=1 (no tiling or loop fusion) (using *_ST1 config dirs) : " "${GROUP_SETS[1]}" printf "%-93s %s\n" "Full tests - NO_QCO without qco (using *_ST2 config dirs) : " "${GROUP_SETS[2]}" # # B. Three different option choices with ORCA2_ICE_PISCES only: # printf "%-93s %s\n" "ORCA2_ICE_PISCES tests - NO_ICB1 without icebergs, with nn_hls=1 (using *_ST3 config dirs) : " "${GROUP_SETS[3]}" printf "%-93s %s\n" "ORCA2_ICE_PISCES tests - NO_ICB2 without icebergs, with nn_hls=2 (using *_ST4 config dirs) : " "${GROUP_SETS[4]}" printf "%-93s %s\n" "ORCA2_ICE_PISCES tests - NO_COLL without collective comms (using *_ST5 config dirs) : " "${GROUP_SETS[5]}" # # A. Full tests for gs in 0 1 2 do for n in `seq 0 1 $(( ${#FULLSET[@]} - 1 ))` do confstr="${FULLSET[$n]}" # run the test echo ./sette.sh ${GROUP_SETS[$gs]} -x "RESTART REPRO CORRUPT" -n "$confstr" ./sette.sh ${GROUP_SETS[$gs]} -x "RESTART REPRO CORRUPT" -n "$confstr" done done # # B. ORCA2_ICE_PISCES special tests for gs in 3 4 5 do # run the test echo ./sette.sh ${GROUP_SETS[$gs]} -x "RESTART REPRO" -n ORCA2_ICE_PISCES ./sette.sh ${GROUP_SETS[$gs]} -x "RESTART REPRO" -n ORCA2_ICE_PISCES done exit will perform all these tests: .. code-block:: bash Full tests - MAIN with default options (using *_ST0 config dirs) : -g 0 -r Full tests - HALO1 with nn_hls=1 (no tiling or loop fusion) (using *_ST1 config dirs) : -e -F -t -g 1 -v HALO1 -r Full tests - NO_QCO without qco (using *_ST2 config dirs) : -q -g 2 -v NO_QCO -r ORCA2_ICE_PISCES tests - NO_ICB1 without icebergs, with nn_hls=1 (using *_ST3 config dirs) : -i -e -F -t -g 3 -v NO_ICB1 -r ORCA2_ICE_PISCES tests - NO_ICB2 without icebergs, with nn_hls=2 (using *_ST4 config dirs) : -i -g 4 -v NO_ICB2 -r ORCA2_ICE_PISCES tests - NO_COLL without collective comms (using *_ST5 config dirs) : -C -g 5 -v NO_COLL -r and create archive structures under: ``MAIN``, ``HALO1``, ``NO_QCO``, ``NO_ICB1``, ``NO_ICB2``, ``NO_COLL`` subdirectories. An new command-line option: ``-r`` has been introduced and used here. It simply prevents the ``sette.sh`` from generating a report on completion. Normally, ``sette.sh`` waits until all jobs have completed and then runs the ``sette_rpt.sh`` script to produce the report. When chaining invocations like this there is no requirement to produce a report between invocations and therefore no need to wait for individual jobs to complete. ``sette_rpt.sh`` can always be invoked directly, see section: `Reporting options`_ for more details. .. _Archive contents: Archive contents ================ ``SETTE`` places copies of selected output files from each test within the relevant subdirectory of the results directory structure. The selection of files is sufficient for simple comparsions between equivalent tests carried out with different code revisions, different run-time options or different compile-time keys. An example full set is: .. code-block:: bash ls -1 NEMO_VALIDATION/HALO1/X86_ARCHER2-Cray/15541/ORCA2_ICE_PISCES/LONG namelist_cfg namelist_ice_cfg namelist_pisces_cfg namelist_top_cfg ocean.output output.namelist.dyn output.namelist.ice output.namelist.pis output.namelist.top run.stat sette_config timing.output tracer.stat but some tests may have fewer files (e.g. not all tests runs will produce a ``tracer.stat`` file and timing information may not have been requested). NOTE: we should add the run.stat.nc files as well The main comparison is performed between ``run.stat`` and ``tracer.stat`` files and any differences will be highlighted. ``ocean.output`` files and the input namelists are included to enable verification of the actual runtime conditions. The output namelists are also included so that accurate reading of the namelists can be verified. A new addition at release 4.2 is the sette_config file which is a record on the options used for this particular invocation of ``sette.sh``. For example: .. code-block:: bash cat ./NEMO_VALIDATION/dev_main_halos/X86_ARCHER2-Cray/21336_0235f28+/GYRE_PISCES/LONG/sette_config Summary of sette environment ---------------------------- requested by the command : ./sette.sh -n GYRE_PISCES -u on branch : dev_main_halos USING_TIMING : yes USING_ICEBERGS : yes USING_EXTRA_HALO : yes USING_TILING : yes USING_COLLECTIVES : yes USING_QCO : yes USING_LOOP_FUSION : yes USING_XIOS : yes USING_MPMD : yes USING_RK3 : no USER_INPUT : no Common compile keys added : key_xios key_loop_fusion key_qco Common compile keys deleted : key_RK3 Compile keys actually used : key_top key_linssh key_xios key_loop_fusion .. _Reporting options: Reporting options ================= As mentioned previously, the reporting function is performed by the ``sette_rpt.sh`` script. This is normally invoked by ``sette.sh`` but can be run at anytime. The script accepts several command-line options which makes it a versatile tool for comparing across different sets of tests: .. code-block:: bash ./sette_rpt.sh -h sette_rpt.sh : display result for the latest change -c COMPILER_name : display result for the specified compiler -r REVISION_number : display sette results for the specified revision (set old for the latest revision available for each config) -R REFERENCE REVISION_number : compare sette results against the specified revision (use to over-ride value set in param.cfg) -v sub_dir : validation sub-directory below NEMO_VALIDATION_DIR -V sub_dir2 : 2nd validation sub-directory below NEMO_VALIDATION_DIR if set the comparison is between two subdirectory trees beneath NEMO_VALIDATION_DIR -u to run sette_rpt.sh without any user interaction ``sette_rpt.sh`` will check all restartability and reproducibility results as well as comparing differences between sets. A stripped down version is available named:``sette_eval.sh`` which can be used for a quick evaluation of differences only. For example: .. code-block:: bash ./sette_eval.sh -V MAIN -v HALO1 -R 14886 Current code is : NEMO/trunk @ r14896 ( last change @ r14886 ) SETTE evaluation for : NEMO/trunk @ r14886 (last changed revision) on X86_ARCHER2-Cray arch file !----result comparison check----! check result differences between : VALID directory : /work/n01/n01/acc/NEMO/2021/midmerge/dev_sette/NEMO_VALIDATION/HALO1 at rev 14886 and REFERENCE directory : /work/n01/n01/acc/NEMO/2021/midmerge/dev_sette/NEMO_VALIDATION/MAIN at rev 14886 GYRE_PISCES run.stat files are identical GYRE_PISCES tracer.stat files are identical ORCA2_ICE_PISCES run.stat files are DIFFERENT (results are different after 71 time steps) ORCA2_ICE_PISCES tracer.stat files are DIFFERENT (results are different after 73 time steps) ORCA2_OFF_PISCES tracer.stat files are DIFFERENT (results are different after 17 time steps) AMM12 run.stat files are identical ORCA2_SAS_ICE run.stat files are identical AGRIF_DEMO run.stat files are identical WED025 run.stat files are identical ISOMIP+ run.stat files are identical VORTEX run.stat files are identical ICE_AGRIF run.stat files are identical OVERFLOW run.stat files are identical LOCK_EXCHANGE run.stat files are identical SWG run.stat files are identical This script also has a "quiet mode" for less verbose output (possibly, more suited for regular monitoring appliactions): .. code-block:: bash ./sette_eval.sh -V MAIN -v HALO1 -R 14886 -q 3 differences from 13 matches. It is also useful when tests have only been run with a few configurations. Take the ``ORCA2_ICE_PISCES`` only tests suggested by ``super_sette.sh``, for example: .. code-block:: bash ./sette_eval.sh -V MAIN -v NO_ICB2 -R 14886 -q 2 differences from 1 matches. 0 missing from REFERENCE 12 missing from VALID The two differences here being some run.stat and tracer.stat differences for ``ORCA2_ICE_PISCES`` at that revision. Reassuringly: .. code-block:: bash ./sette_eval.sh -V NO_ICB2 -v NO_ICB1 -R 14886 -q 0 differences from 1 matches. 12 missing from REFERENCE 12 missing from VALID and: .. code-block:: bash ./sette_eval.sh -V MAIN -v NO_COLL -R 14886 -q 0 differences from 1 matches. 0 missing from REFERENCE 12 missing from VALID Keeping track of archived results ================================= It is apparent that with all this flexibility comes the risk of losing track of which revisions have been tested with which configurations. ``SETTE`` now includes a helper script which can table the options available for comparison. E.g.: .. code-block:: bash ./sette_list_avail_rev.sh Compiler used is : X86_ARCHER2-Cray List of all avail. rev. in :/work/n01/n01/acc/NEMO/GIT/nemo/sette/NEMO_VALIDATION/MAIN/ is : 21334_522ad4c+ 21335_440ad90+ 21336_0235f28+ 21336_416860f+ Availability for each config.: ------------------------------ GYRE_PISCES : ------------ 21335_440ad90+ 21336_0235f28+ 21336_416860f+ ORCA2_ICE_PISCES : ------------ 21335_440ad90+ 21336_0235f28+ 21336_416860f+ ORCA2_OFF_PISCES : ------------ ------------ 21336_0235f28+ ------------ AMM12 : ------------ ------------ 21336_0235f28+ ------------ WED025 : ------------ ------------ 21336_0235f28+ ------------ ORCA2_ICE_OBS : ------------ ------------ 21336_0235f28+ ------------ ORCA2_SAS_ICE : ------------ ------------ 21336_0235f28+ ------------ AGRIF_DEMO : ------------ ------------ 21336_0235f28+ ------------ SWG : ------------ ------------ ------------ ------------ ISOMIP+ : 21334_522ad4c+ ------------ 21336_0235f28+ ------------ OVERFLOW : ------------ ------------ 21336_0235f28+ ------------ LOCK_EXCHANGE : ------------ ------------ 21336_0235f28+ ------------ VORTEX : ------------ ------------ 21336_0235f28+ ------------ ICE_AGRIF : ------------ ------------ 21336_0235f28+ ------------ By default, this lists the options in the MAIN sub-directory but other parts of the directory tree can be examined using command-line arguments: .. code-block:: bash sette_list_avail_rev.sh : list all sette directory and available revisions created with the compiler specified in param.cfg or in the startup file) -c COMPILER_name : list all sette directory and available revisions created with the compiler specified -v sub_dir : validation sub-directory below NEMO_VALIDATION_DIR To add a new configuration ========================== 1. creates a new ``input_NEW_CONFIG.cfg`` if you need tar file (if you use same tar file of GYRE, ORCA2_LIM or ORCA2_LIM_PISCES you can use it) 2. add a bloc in one of the ``sette_reference-configuration.sh`` or ``sette_test-cases.sh`` script 3. add your configuration to the list in ``param.cfg`` .. image:: _static/sette_rpt.png :align: center