Deep Neural Networks in Kaldi

Introduction

Deep Neural Networks (DNNs) are the latest hot topic in speech recognition. Since around 2010 many papers have been published in this area, and some of the largest companies (e.g. Google, Microsoft) are starting to use DNNs in their production systems. An active area of research like this is difficult for a toolkit like Kaldi to support well, because the state of the art changes constantly which means code changes are required to keep up, and architectural decisions may need to be rethought.

We currently have three separate codebases for deep neural nets in Kaldi. All are still active in the sense that the up-to-date recipes refer to all of them. The first one ("nnet1"( is located in code subdirectories nnet/ and nnetbin/, and is primarily maintained by Karel Vesely. The second is located in code subdirectories nnet2/ and nnet2bin/, and is primarily maintained by Daniel Povey (this code was originally based on an earlier version of Karel's code, but it has been extensively rewritten). The third is located in code subdirectories nnet3/ and nnet3bin/, and Dan's previous work on nnet2 will shift to the nnet3 setup.

In the example directories such as egs/wsj/s5/, egs/rm/s5, egs/swbd/s5 and egs/hkust/s5b, neural net example scripts can be found. Karel's example scripts can be found in local/nnet/run_dnn.sh, and Dan's example scripts can be found in local/run_nnet2.sh. Before running those scripts, the first stages of ``run.sh'' in those directories must be run in order to build the systems used for alignment.

Regarding which of the setups you should use:

  • Karel's setup (nnet1) supports training on a single GPU card, which allows the implementation to be simpler and relatively easy to modify.
  • Dan's setup (nnet2) is more flexible in how you can train: it supports using multiple GPUs, or multiple CPU's each with multiple threads. Multiple GPU's is the recommended setup. They don't have to all be on the same machine. Both setups give commensurate results.
  • The ``new'' setup, nnet3 (nnet3) is intended (at least, by Dan) to be the recommended path going forward; how functional it is may depend on when you read this.

Between the setups there are many differences in the recipes. For example, Karel's setup uses pre-training but Dan's setup does not; Karel's setup uses early stopping using a validation set but Dan's setup uses a fixed number of epochs and averages the parameters over the last few epochs of training. Most other details of the training (nonlinearity types, learning rate schedules, network topology, input features etc.) also differ.

The best published descriptions of the DNN setups are:

The setups use incompatible DNN formats, while there is a converter of Karel's network into Dan's format Conversion of a DNN model between nnet1 -> nnet2.