Skip to content

Experiment Directory

Each time you run an experiment, SURF creates a unique experiment directory containing all the experiment-specific outputs such as configuration files, simulation outputs, and postprocessed results.

Note

In some cases, it can be useful to configure SURF to mount an existing experiment directory instead of creating a new one. To learn more, see Resume an existing experiment.

In this page, you will learn more about the experiment directory layout and its subfolder contents.

Directory Layout

graph LR
    base[🏠&nbsp Base Directory] --> A[📁&nbsp Experiment Directory]
    A --> B[📁&nbsp configuration/]
    A --> C[📁&nbsp logfiles/]
    A --> D[📁&nbsp postprocessing/]
    A --> E[📁&nbsp preprocessing/]

    D --> D1[📁&nbsp YYYYMMDD/]

    E --> E1[📁&nbsp atmosphere_sbc/]
    E --> E2[📁&nbsp meshing/]
    E2 --> E2a[📁&nbsp bathymetry/]
    E2 --> E2b[📁&nbsp coastline/]
    E2 --> E2c[📁&nbsp mesh/]
    E2 --> E2d[📁&nbsp river/]
    E --> E3[📁&nbsp ocean_ic/]
    E --> E4[📁&nbsp ocean_obc/]
    E --> E5[📁&nbsp river_bc/]

    A --> F[📁&nbsp simulation/]
    F --> F1[📁&nbsp YYYYMMDD/]
    F --> F8[📁&nbsp xml/]

    %% Add links
    click base "#base-directory"
    click A "#experiment-directory"
    click B "#configuration"
    click C "#logfiles"
    click D "#postprocessing"
    click E "#preprocessing"
    click F "#simulation"

Details

Base Directory

Click here to expand...

The Base Directory corresponds to the output-dir directory chosen when launching SURF-SHYFEM with the CLI:

1
surf_shyfem --output-dir [/BASE/DIRECTORY/]

For example, in your first experiment you configured the Base Directory to: $SURF_SHYFEM_DIR.

The Base Directory can be any location in your filesystem, also an external volume (e.g. a USB drive). The only requirement is that you must have permission to write in this directory.

Internally, SURF mounts the Base Directory as a Docker volume.

Experiment Directory

Click here to expand...

The Experiment Directory is created inside the Base Directory, and is named after the 🔗Experiment name parameter in the SURF configuration file.

For example, in your first experiment you configured the Experiment Directory to: bay_madagascar.

When the experiment is launched, if the experiment directory does not exist, it is created inside the Base Directory. If instead it already exists, the execution depends on the value of the Overwrite parameter in the configuration file. If the flag is set to true, the experiment directory is overwritten (erased and recreated from scratch), otherwise the folder creation will be simply skipped. See also Resume an existing experiment.

configuration/

Click here to expand...

This folder contains parsed versions of the SURF configuration files. It is mostly meant for internal use, you should not need to interact with it.

logfiles/

Click here to expand...

This folder contains a separate log file for each step of the SURF workflow.

This is particularly useful for monitoring progress and debugging issues with your experiment.

For example, if your experiment failed at the Grid Generation step, you can check the logfiles/STEP_1_Grid_Generation.log file to see what went wrong.

preprocessing/

Click here to expand...

This folder includes all data that is downloaded and produced by SURF before launching the simulation. It is mostly meant for internal use, although it may be sometimes useful to inspect its contents for debugging purposes.

Each subfolder corresponds to a specific preprocessing step:

preprocessing/
├── atmosphere_sbc/     —> Preparation of atmospheric surface boundary conditions
├── meshing/            —> Generation of computational meshes and related datasets (Bathymetric data processing, Coastline data processing and grid creation)
├── ocean_ic/           —> Processing of ocean initial conditions
├── ocean_obc/          —> Open boundary conditions for the ocean model
├── river_bc/           —> River boundary conditions for the ocean model

The internal structure of each subfolder (e.g. ocean_obc/) contains the following subfolders: 

ocean_obc/
├── extrapolate         —> Sea-over-land extrapolation
├── input               —> Input data download
├── interpolate         —> Interpolation of 1D open boundary conditions to the 3D grid
├── reformat            —> Reformatting of input data to SURF internal format
└── tidal_forcing       —> Application of tidal correction to sea surface height values

simulation/

Click here to expand...

This folder contains the raw outputs of the numerical simulation (e.g. log files and NetCDF files produced by SHYFEM-MPI).

It contains a list of subfolders, one for each simulation date, with names of the form YYYYMMDD_YYYYMMDD. For example:

  • 20210403_20210404/: Output for the day 03 Apr 2021 00:00:00 to 04 Apr 2021 00:00:00
  • 20210404_20210405/: Output for the day 04 Apr 2021 00:00:00 to 05 Apr 2021 00:00:00

Additionally, it also contains the namelist.str file, a configuration file automatically generated within the SURF workflow that contains all the necessary input parameters for running the SHYFEM-MPI model (see the SHYFEM-MPI Configuration section).

postprocessing/

Click here to expand...

This folder contains the postprocessed outputs of the numerical simulation (e.g. static images and GIFs).

It is organized in subfolders, one for each simulation date, similar to the simulation/ folder.

Each subfolder in turn contains different sections for each output format:

  • png/: PNG images
  • gif/: GIFs
  • pdf/: PDF files

To learn how to customize your postprocessing output, see the Customize the output figures section.