Skip to content

Experiment Directory

Each time you run an experiment, SURF creates a unique experiment directory containing all the experiment-specific outputs such as configuration files, simulation outputs, and postprocessed results.

Note

In some cases, it can be useful to configure SURF to mount an existing experiment directory instead of creating a new one. To learn more, see Resume an existing experiment.

In this page, you will learn more about the experiment directory layout and its subfolder contents.

Directory Layout

graph LR
    base[🏠&nbsp Base Directory] --> A[📁&nbsp Experiment Directory]
    A --> B[📁&nbsp configuration/]
    A --> C[📁&nbsp logfiles/]
    A --> D[📁&nbsp postprocessing/]
    A --> E[📁&nbsp preprocessing/]

    D --> D1[📁&nbsp YYYYMMDD/]

    E --> E1[📁&nbsp atmosphere_sbc/]
    E --> E2[📁&nbsp meshing/]
    E2 --> E2a[📁&nbsp bathymetry/]
    E2 --> E2b[📁&nbsp coastline/]
    E2 --> E2c[📁&nbsp mesh/]
    E --> E3[📁&nbsp ocean_ic/]
    E --> E4[📁&nbsp ocean_obc/]
    E --> E5[📁&nbsp tides_obc/]

    A --> F[📁&nbsp simulation/]
    F --> F1[📁&nbsp YYYYMMDD/]
    F --> F8[📁&nbsp xml/]

    %% Add links
    click base "#base-directory"
    click A "#experiment-directory"
    click B "#configuration"
    click C "#logfiles"
    click D "#postprocessing"
    click E "#preprocessing"
    click F "#simulation"

Details

Base Directory

Click here to expand...

The Base Directory corresponds to the output-dir directory chosen when launching SURF-NEMO with the CLI:

1
surf_nemo --output-dir [/BASE/DIRECTORY/]

For example, in your first experiment you configured the Base Directory to: $SURF_NEMO_DIR.

The Base Directory can be any location in your filesystem, also an external volume (e.g. a USB drive). The only requirement is that you must have permission to write in this directory.

Internally, SURF mounts the Base Directory as a Docker volume.

Experiment Directory

Click here to expand...

The Experiment Directory is created inside the Base Directory, and is named after the 🔗Experiment name parameter in the SURF configuration file.

For example, in your first experiment you configured the Experiment Directory to: gulf_taranto.

When the experiment is launched, if the experiment directory does not exist, it is created inside the Base Directory. If instead it already exists, the execution depends on the value of the Overwrite parameter in the configuration file. If the flag is set to true, the experiment directory is overwritten (erased and recreated from scratch), otherwise the folder creation will be simply skipped. See also Resume an existing experiment.

configuration/

Click here to expand...

This folder contains parsed versions of the SURF configuration files. It is mostly meant for internal use, you should not need to interact with it.

logfiles/

Click here to expand...

This folder contains a separate log file for each step of the SURF workflow.

This is particularly useful for monitoring progress and debugging issues with your experiment.

For example, if your experiment failed at the Grid Generation step, you can check the logfiles/STEP_1_Grid_Generation.log file to see what went wrong.

preprocessing/

Click here to expand...

This folder includes all data that is downloaded and produced by SURF before launching the simulation. It is mostly meant for internal use, although it may be sometimes useful to inspect its contents for debugging purposes.

Each subfolder corresponds to a specific preprocessing step:

preprocessing/
├── atmosphere_sbc/     —> Preparation of atmospheric surface boundary conditions
├── meshing/            —> Generation of computational meshes and related datasets (Bathymetric data processing, Coastline data processing and grid creation)
├── ocean_ic/           —> Processing of ocean initial conditions
├── ocean_obc/          —> Open boundary conditions for the ocean model
└── tides_obc/          —> Preparation of tidal boundary conditions

The internal structure of each subfolder (e.g. ocean_obc/) contains the following subfolders: 

ocean_obc/
├── 1d                  —> 1D open boundary conditions (i.e. open boundary conditions extracted as a 1D array along the downscaled domain boundary)
├── extrapolate         —> Sea-over-land extrapolation
├── input               —> Input data download
├── interpolate         —> Interpolation of 1D open boundary conditions to the 3D grid
└── reformat            —> Reformatting of input data to SURF internal format

simulation/

Click here to expand...

This folder contains the raw outputs of the numerical simulation (e.g. log files and NetCDF files produced by NEMO).

It contains a list of subfolders, one for each simulation date, with names of the form YYYYMMDD_YYYYMMDD. For example:

  • 20141004_20141005/: Output for the day 04 Oct 2014 00:00:00 to 05 Oct 2014 00:00:00
  • 20141005_20141006/: Output for the day 05 Oct 2014 00:00:00 to 06 Oct 2014 00:00:00

Additionally, it also contains an xml/ folder, which contains the XIOS configuration in XML format (for the XIOS version and configuration see the NEMO Configuration section).

if you want to know more about the NEMO outputs and XIOS configuration files, please refer to the NEMO documentation.

postprocessing/

Click here to expand...

This folder contains the postprocessed outputs of the numerical simulation (e.g. static images and GIFs).

It is organized in subfolders, one for each simulation date, similar to the simulation/ folder.

Each subfolder in turn contains different sections for each output format:

  • png/: PNG images
  • gif/: GIFs
  • pdf/: PDF files

To learn how to customize your postprocessing output, see the Customize the output figures section.