NeMo Audio Configuration Files

NeMo-Speech

This section describes the NeMo configuration file setup that is specific to models in the audio collection. For general information about how to set up and run experiments that is common to all NeMo models (e.g. Experiment Manager and PyTorch Lightning trainer parameters), see the

NeMo Models

section.

The model section of the NeMo audio configuration files generally requires information about the dataset(s) being used, parameters for any augmentation being performed, as well as the model architecture specification.

Example configuration files for all of the NeMo audio models can be found in the

config directory of the examples

NeMo Dataset Configuration

Training, validation, and test parameters are specified using the model.train_ds, model.validation_ds, and model.test_ds sections in the configuration file, respectively. Depending on the task, there may be arguments specifying the sample rate or duration of the loaded audio examples. Some fields can be left out and specified via the command-line at runtime. Refer to the

Dataset Processing Classes

section of the API for a list of datasets classes and their respective parameters. An example train, validation and test datasets can be configured as follows:

model:sample_rate:16000skip_nan_grad:falsetrain_ds:manifest_filepath:???input_key:audio_filepath# key of the input signal path in the manifesttarget_key:target_filepath# key of the target signal path in the manifesttarget_channel_selector:0# target signal is the first channel from files in target_keyaudio_duration:4.0# in seconds, audio segment duration for trainingrandom_offset:true# if the file is longer than audio_duration, use random offset to select a subsegmentmin_duration:${model.train_ds.audio_duration}batch_size:64# batch size may be increased based on the available memoryshuffle:truenum_workers:8pin_memory:truevalidation_ds:manifest_filepath:???input_key:audio_filepath# key of the input signal path in the manifesttarget_key:target_filepath# key of the target signal path in the manifesttarget_channel_selector:0# target signal is the first channel from files in target_keybatch_size:64# batch size may be increased based on the available memoryshuffle:falsenum_workers:4pin_memory:truetest_ds:manifest_filepath:???input_key:audio_filepath# key of the input signal path in the manifesttarget_key:target_filepath# key of the target signal path in the manifesttarget_channel_selector:0# target signal is the first channel from files in target_keybatch_size:1# batch size may be increased based on the available memoryshuffle:falsenum_workers:4pin_memory:trueMore information about online augmentation can found in the

masking example configuration

Lhotse Dataset Configuration

Lhotse CutSet

An example train dataset in Lhotse CutSet format can be configured as follows:

train_ds:use_lhotse:true# enable Lhotse data loadercuts_path:???# path to Lhotse cuts manifest with input signals and the corresponding target signals (target signals should be in the custom "target_recording" field)truncate_duration:4.00# truncate audio to 4 secondstruncate_offset_type:random# if the file is longer than truncate_duration, use random offset to select a subsegmentbatch_size:64# batch size may be increased based on the available memoryshuffle:truenum_workers:8pin_memory:trueLhotse CutSet with Online Augmentation

An example train dataset in Lhotse CutSet format using online augmentation with room impulse response (RIR) convolution and additive noise can be configured as follows:

train_ds:use_lhotse:true# enable Lhotse data loadercuts_path:???# path to Lhotse cuts manifest with speech signals for augmentation (including custom "target_recording" field with the same signals)truncate_duration:4.00# truncate audio to 4 secondstruncate_offset_type:random# if the file is longer than truncate_duration, use random offset to select a subsegmentbatch_size:64# batch size may be increased based on the available memoryshuffle:truenum_workers:8pin_memory:truerir_enabled:true# enable room impulse response augmentationrir_path:???# path to Lhotse recordings manifest with room impulse response signalsnoise_path:???# path to Lhotse cuts manifest with noise signalsA configuration file with Lhotse online augmentation can found in the

online augmentation example configuration

. More information about the online augmentation can be found in the

tutorial notebook

Lhotse Shar

An example train dataset in Lhotse shar format can be configured as follows:

train_ds:shar_path:???use_lhotse:truetruncate_duration:4.00# truncate audio to 4 secondstruncate_offset_type:randombatch_size:8# batch size may be increased based on the available memoryshuffle:truenum_workers:8pin_memory:trueA configuration file with Lhotse shar format can found in the

SSL pretraining example configuration

Dataset Reweighting with Temperature

When combining multiple datasets using nested input_cfg groups, you can control the sampling distribution using the reweight_temperature parameter. This feature allows you to balance dataset sampling without manually recalculating weights when adding or removing datasets.

The temperature scaling formula is:

\[\hat{w}_i = \frac{w_i^{\tau}}{\sum_{j} w_j^{\tau}}\]

where $w_i$ is the original weight of dataset $i$, $\tau$ is the temperature, and $\hat{w}_i$ is the normalized sampling probability.

How Temperature Works:

temperature=1.0: Preserves original weight ratios (neutral, no reweighting)

temperature=0.0: Equalizes all datasets (each gets equal probability regardless of original weights)

0<temperature<1.0: Over-samples smaller datasets relative to larger ones

temperature>1.0: Amplifies differences between dataset weights

Configuration Options:

The reweight_temperature parameter accepts two formats:

Scalar value (applied to all nesting levels, warning logged):

train_ds:use_lhotse:truereweight_temperature:0.5# Applied to all levels, warning loggedinput_cfg:-type:groupinput_cfg:-type:lhotse_sharshar_path:/path/to/dataset1weight:900-type:lhotse_sharshar_path:/path/to/dataset2weight:100-type:lhotse_sharshar_path:/path/to/dataset3weight:200-type:nemo_tarredmanifest_filepath:/path/to/dataset4/manifest.jsontarred_audio_filepath:/path/to/dataset4/audio.tarweight:300List matching maximum nesting depth (one temperature per level):

train_ds:use_lhotse:truereweight_temperature:[1.0,0.0]# Level 1: preserve ratios, Level 2: equalizeinput_cfg:-type:groupweight:0.7input_cfg:-type:lhotse_sharshar_path:/path/to/dataset1weight:600-type:lhotse_sharshar_path:/path/to/dataset2weight:400-type:groupweight:0.3input_cfg:-type:lhotse_sharshar_path:/path/to/dataset3weight:100Note

If reweight_temperature is provided as a list, its length must exactly match the maximum nesting depth of input_cfg. A mismatch (too few or too many values) raises a ValueError. Use a scalar value instead if you want the same temperature applied uniformly to all levels.

Maximum Nesting Depth Calculation:

The maximum nesting depth is calculated as the maximum depth of input_cfg keys in the configuration. Sibling groups at the same level share the same temperature value.

# This has maximum nesting depth = 2input_cfg:# Level 1-type:groupinput_cfg:# Level 2-type:lhotse_shar-type:group# Same level as above (sibling)input_cfg:# Level 2 (same as above)-type:lhotse_sharWhen input_cfg is overridden via CLI to a YAML file path (e.g. model.train_ds.input_cfg=train_all.yaml), the depth calculation loads the referenced file and traverses its contents to count nested input_cfg keys. This also works with multi-level file references:

# train_all.yaml (referenced via input_cfg=train_all.yaml)-type:groupweight:100input_cfg:${oc.env:MANIFEST_ROOT}/train_en.yaml# resolved at runtime-type:groupweight:200input_cfg:${oc.env:MANIFEST_ROOT}/train_de.yamlNote

Paths containing OmegaConf interpolations (e.g. ${oc.env:MANIFEST_ROOT}) cannot be resolved during depth counting – they are resolved later at runtime by OmegaConf.create(). Such paths are treated as a single additional nesting level.

Example: Balancing Multiple Task Groups

train_ds:use_lhotse:truereweight_temperature:[1.0,0.0]# Level 1: Preserve task ratios, Level 2: Equalize within tasksinput_cfg:-type:groupweight:0.7tags:task:asrinput_cfg:-type:nemo_tarredmanifest_filepath:/path/to/asr1/manifest.jsontarred_audio_filepath:/path/to/asr1/audio.tarweight:600# Large dataset-type:nemo_tarredmanifest_filepath:/path/to/asr2/manifest.jsontarred_audio_filepath:/path/to/asr2/audio.tarweight:100# Small dataset (will be upsampled with temp=0.0)-type:groupweight:0.3tags:task:astinput_cfg:-type:nemo_tarredmanifest_filepath:/path/to/ast1/manifest.jsontarred_audio_filepath:/path/to/ast1/audio.tarweight:50-type:nemo_tarredmanifest_filepath:/path/to/ast2/manifest.jsontarred_audio_filepath:/path/to/ast2/audio.tarweight:200In this example:

Level 1 temperature is 1.0: The 70/30 split between ASR and AST groups is preserved

Level 2 temperature is 0.0: Within each group, all datasets are sampled equally regardless of their original weights

Model Architecture Configuration

Each configuration file should describe the model architecture being used for the experiment. An example of a simple predictive model configuration is shown below:

model:type:predictivesample_rate:16000skip_nan_grad:falsenum_outputs:1normalize_input:true# normalize the input signal to 0dBFStrain_ds:manifest_filepath:???input_key:noisy_filepathtarget_key:clean_filepathaudio_duration:2.00# trim audio to 2 secondsrandom_offset:truenormalization_signal:input_signalbatch_size:8# batch size may be increased based on the available memoryshuffle:truenum_workers:8pin_memory:truevalidation_ds:manifest_filepath:???input_key:noisy_filepathtarget_key:clean_filepathbatch_size:8shuffle:falsenum_workers:4pin_memory:trueencoder:_target_:nemo.collections.audio.modules.transforms.AudioToSpectrogramfft_length:510# Number of subbands in the STFT = fft_length // 2 + 1 = 256hop_length:128magnitude_power:0.5scale:0.33decoder:_target_:nemo.collections.audio.modules.transforms.SpectrogramToAudiofft_length:${model.encoder.fft_length}hop_length:${model.encoder.hop_length}magnitude_power:${model.encoder.magnitude_power}scale:${model.encoder.scale}estimator:_target_:nemo.collections.audio.parts.submodules.ncsnpp.SpectrogramNoiseConditionalScoreNetworkPlusPlusin_channels:1# single-channel noisy inputout_channels:1# single-channel estimatenum_res_blocks:3# increased number of res blockspad_time_to:64# pad to 64 frames for the time dimensionpad_dimension_to:0# no padding in the frequency dimensionloss:_target_:nemo.collections.audio.losses.MSELoss# computed in the time domainmetrics:val:sisdr:# output SI-SDR_target_:torchmetrics.audio.ScaleInvariantSignalDistortionRatiooptim:name:adamlr:1e-4# optimizer argumentsbetas:[0.9,0.999]weight_decay:0.0Complete configuration file can found in the

example configuration

Finetuning Configuration

All scripts support easy finetuning by partially/fully loading the pretrained weights from a checkpoint into the currently instantiated model. Note that the currently instantiated model should have parameters that match the pre-trained checkpoint so the weights may load properly.

Pre-trained weights can be provided by:

Providing a path to a NeMo model (via init_from_nemo_model)

Providing a name of a pretrained NeMo model (which will be downloaded via the cloud) (via init_from_pretrained_model)

Training from scratch

A model can be trained from scratch using the following command:

pythonexamples/audio/audio_to_audio_train.py\--config-path=<pathtodirofconfigs> --config-name=<nameofconfigwithout.yaml>)\model.train_ds.manifest_filepath="<path to manifest file>"\model.validation_ds.manifest_filepath="<path to manifest file>"\trainer.devices=1\trainer.accelerator='gpu'\trainer.max_epochs=50Fine-tuning via a NeMo model

A model can be finetuned from an existing NeMo model using the following command:

A model can be finetuned from an pre-trained NeMo model using the following command: