Quickstart

The following quickstart guide provides a short introduction to autrainer and the creation of simple training experiments.

First Experiment

To get started, create a new directory and navigate to it:

mkdir autrainer_example && cd autrainer_example

Next, create a new empty autrainer project using the following configuration management CLI command:

autrainer create --empty

Alternatively, use the following configuration management CLI wrapper function:

import autrainer.cli # the import is omitted in the following examples for brevity

autrainer.cli.create(empty=True)

This will create the configuration directory structure and the main configuration (conf/config.yaml) file with default values:

conf/config.yaml

defaults:
  - _autrainer_
  - _self_

results_dir: results
experiment_id: default
iterations: 5

hydra:
  sweeper:
    params:
      +seed: 1
      +batch_size: 32
      +learning_rate: 0.001
      dataset: ToyTabular-C
      model: ToyFFNN
      optimizer: Adam

Now, run the following training command to train the model:

autrainer train

Alternatively, use the following training CLI wrapper function:

autrainer.cli.train() # the train function is omitted in the following examples for brevity

This will train the default ToyFFNN feed-forward neural network (FFNN) on the default ToyTabular-C classification dataset with tabular data (ToyDataset) and output the training results to the results/default/ directory.

Custom Model Configuration

The first experiment uses the default ToyFFNN model with the following configuration having 2 hidden layers:

conf/model/ToyFFNN.yaml

id: ToyFFNN
_target_: autrainer.models.FFNN
input_size: 64
hidden_size: 64
num_layers: 2

transform:
  type: tabular

To create another configuration for the FFNN model with 3 hidden layers, create a new configuration file in the conf/model/ directory:

conf/model/Three-Layer-FFNN.yaml

id: Three-Layer-FFNN
_target_: autrainer.models.FFNN
input_size: 64
hidden_size: 64
num_layers: 3 # 3 hidden layers

transform:
  type: tabular

Next, update the main configuration (conf/config.yaml) file to use the new model configuration:

conf/config.yaml

defaults:
  - _autrainer_
  - _self_

results_dir: results
experiment_id: default
iterations: 5

hydra:
  sweeper:
    params:
      +seed: 1
      +batch_size: 32
      +learning_rate: 0.001
      dataset: ToyTabular-C
      model: Three-Layer-FFNN # 3 hidden layers
      optimizer: Adam

Now, run the following training command to train the model with 3 hidden layers:

autrainer train

Grid Search Configuration

To perform a grid search over multiple multiple configurations defined in the params, update the main configuration (conf/config.yaml) to include multiple values separated by a comma.

The following configuration performs a grid search over the default FFNN model with 2 and 3 hidden layers as well as 3 different seeds:

conf/config.yaml

defaults:
  - _autrainer_
  - _self_

results_dir: results
experiment_id: default
iterations: 5

hydra:
  sweeper:
    params:
      +seed: 1, 2, 3 # 3 seeds to compare
      +batch_size: 32
      +learning_rate: 0.001
      dataset: ToyTabular-C
      model: ToyFFNN, Three-Layer-FFNN # 2 models to compare
      optimizer: Adam

Now, run the following training command to train the models with 2 and 3 hidden layers and 3 different seeds:

autrainer train

By default, a grid search is performed sequentially. Hydra allows the use of different launcher plugins to perform parallel grid searches.

Note

If a run already exists in the same experiment and has been completed successfully, then it will be skipped. This may be the case for both the default and custom model configurations with seed 1 if they have already been trained in the previous examples.

To compare the results of the individual runs as well as averaged across seeds, run the following postprocessing command:

autrainer postprocess results default --aggregate seed

Alternatively, use the following postprocessing CLI wrapper function:

autrainer.cli.postprocess(
    results_dir="results",
    experiment_id="default",
    aggregate=[["seed"]],
)

Spectrogram Classification

To train a Cnn10 model on an audio dataset such as DCASE2016Task1, update the main configuration (conf/config.yaml) file:

conf/config.yaml

defaults:
  - _autrainer_
  - _self_

results_dir: results
experiment_id: spectrogram
iterations: 5

hydra:
  sweeper:
    params:
      +seed: 1
      +batch_size: 32
      +learning_rate: 0.001
      dataset: DCASE2016Task1-32k
      model: Cnn10-32k-T
      optimizer: Adam

For the Cnn10 model, the following configuration is used:

conf/model/Cnn10-32k-T.yaml

id: Cnn10-32k-T
_target_: autrainer.models.Cnn10
transfer: https://zenodo.org/records/3987831/files/Cnn10_mAP%3D0.380.pth

transform:
  type: grayscale
  base:
    - autrainer.transforms.Normalize: null

The ending 32k-T indicates that the model using transfer learning and has been pretrained with a sample rate of 32 kHz.

Tip

To discover all available default configurations for e.g. different models, the configuration management CLI, the configuration management CLI wrapper, and the models documentation can be used.

For the DCASE2016Task1 dataset, the following configuration is used:

conf/dataset/DCASE2016Task1-32k.yaml

id: DCASE2016Task1-32k
_target_: autrainer.datasets.DCASE2016Task1

fold: 1

path: data/DCASE2016
features_subdir: log_mel_32k
index_column: filename
target_column: scene_label
file_type: npy
file_handler: autrainer.datasets.utils.NumpyFileHandler

criterion: autrainer.criterions.BalancedCrossEntropyLoss
metrics: 
  - autrainer.metrics.Accuracy
  - autrainer.metrics.UAR
  - autrainer.metrics.F1
tracking_metric: autrainer.metrics.Accuracy

transform:
  type: grayscale

The ending 32k indicates that the dataset has a sample rate of 32 kHz and provides log-Mel spectrograms instead of raw audio.

To avoid race conditions when using Launcher Plugins that may run multiple training jobs in parallel, the following preprocessing command is used to fetch and download the model weights and the raw audio files of the dataset:

autrainer fetch

Alternatively, use the following preprocessing CLI wrapper function:

autrainer.cli.fetch()

As the dataset uses log-Mel spectrograms instead of the raw audio files downloaded in the previous step, the following preprocessing command is used to preprocess and extract the features from the raw audio files:

autrainer preprocess

Alternatively, use the following preprocessing CLI wrapper function:

autrainer.cli.preprocess()

Now, run the following training command to train the model on the audio dataset:

autrainer train

Training Duration & Step-based Training

By default, autrainer uses epoch-based training, where the iterations correspond to the number of epochs. To change the training duration of the spectrogram classification model, increase the number of iterations in the main configuration (conf/config.yaml) file.

However, to use step-based training instead of epoch-based training, set the training_type to step.

The following configuration trains the spectrogram classification model for a total of 1000 steps with step-based training, evaluating every 100 steps, saving the states every 200 steps, and without displaying a progress bar:

conf/config.yaml

defaults:
  - _autrainer_
  - _self_

results_dir: results
experiment_id: spectrogram_step

training_type: step
iterations: 1000
eval_frequency: 100
save_frequency: 200
progress_bar: false

hydra:
  sweeper:
    params:
      +seed: 1
      +batch_size: 32
      +learning_rate: 0.001
      dataset: DCASE2016Task1-32k
      model: Cnn10-32k-T
      optimizer: Adam

Now, run the following training command to train the model on the audio dataset for 1000 steps:

autrainer train

Filtering Configurations

By default, autrainer filters out any configurations that have already been trained and exist in the same experiment using the hydra-filter-sweeper plugin with the following filters that are implicitly set in the _autrainer_.yaml defaults file:

conf/config.yaml

    filters:
      - type: exists
        path: metrics.csv

To filter out unwanted configurations and exclude them from training, the hydra-filter-sweeper plugin can be used as the Hydra sweeper plugin. hydra-filter-sweeper allows to specify a list of filters to exclude configurations based on their attributes.

The following configuration expands the grid search configuration and adds a filter that excludes any seed greater than 2 for the Three-Layer-FFNN model:

conf/config.yaml

defaults:
  - _autrainer_
  - _self_

results_dir: results
experiment_id: default
iterations: 5

hydra:
  sweeper:
    params:
      +seed: 1, 2, 3
      +batch_size: 32
      +learning_rate: 0.001
      dataset: ToyTabular-C
      model: ToyFFNN, Three-Layer-FFNN
      optimizer: Adam
    filters:
      - type: exists
        path: metrics.csv
      - type: expr
        expr: model.id == "Three-Layer-FFNN" and seed > 2

Note

If the filters attribute is overridden in the main configuration (conf/config.yaml) file, then the default filters are not applied. To still filter out configurations that have already been trained, the following default filter should still be included:

conf/config.yaml

    filters:
      - type: exists
        path: metrics.csv

Now, run the following training command to train the ToyFFNN with 3 seeds and the Three-Layer-FFNN with 2 seeds:

autrainer train

Next Steps

For more information on creating configurations, refer to the Hydra configurations as well as the Hydra documentation.

To create custom implementations alongside configurations, refer to the tutorials.