NnetUpdater Class Reference

#include <nnet-update.h>

Collaboration diagram for NnetUpdater:

Public Member Functions
	NnetUpdater (const Nnet &nnet, Nnet *nnet_to_update)
double	ComputeForMinibatch (const std::vector< NnetExample > &data, double *tot_accuracy)
	Does the entire forward and backward computation for this minbatch. More...
double	ComputeForMinibatch (const std::vector< NnetExample > &data, Matrix< BaseFloat > *formatted_data, double *tot_accuracy)
	This version of ComputeForMinibatch is used when you have already called the function FormatNnetInput (defined below) to format your data as a single matrix. More...
void	GetOutput (CuMatrix< BaseFloat > *output)

Protected Member Functions
void	Propagate ()
void	FormatInput (const std::vector< NnetExample > &data)
	Formats the input as a single matrix and sets the size of forward_data_, and sets up chunk_info_out_. More...
double	ComputeObjfAndDeriv (const std::vector< NnetExample > &data, CuMatrix< BaseFloat > *deriv, double *tot_accuracy=NULL) const
	Computes objective function and derivative at output layer, but does not do the backprop [for that, see Backprop()]. More...
void	Backprop (CuMatrix< BaseFloat > *deriv) const
	Backprop must be called after ComputeObjfAndDeriv. More...

Private Member Functions
double	ComputeTotAccuracy (const std::vector< NnetExample > &data) const

Private Attributes
const Nnet &	nnet_
Nnet *	nnet_to_update_
int32	num_chunks_
std::vector< ChunkInfo >	chunk_info_out_
std::vector< CuMatrix< BaseFloat > >	forward_data_

Friends
class	NnetEnsembleTrainer

Detailed Description

Definition at line 46 of file nnet-update.h.

Constructor & Destructor Documentation

NnetUpdater()

NnetUpdater	(	const Nnet &	nnet,
		Nnet *	nnet_to_update
	)

Definition at line 28 of file nnet-update.cc.

                                              :
    nnet_(nnet), nnet_to_update_(nnet_to_update) {
}

Member Function Documentation

Backprop()

void Backprop ( CuMatrix< BaseFloat > *  deriv ) const

protected

Backprop must be called after ComputeObjfAndDeriv.

Does the backpropagation; "nnet_to_update_" is updated. Note: "deriv" will contain, at input, the derivative w.r.t. the output layer (as computed by ComputeObjfAndDeriv), but will be used as a temporary variable by this function.

Definition at line 188 of file nnet-update.cc.

References Component::Backprop(), NnetUpdater::chunk_info_out_, Nnet::FirstUpdatableComponent(), NnetUpdater::forward_data_, Nnet::GetComponent(), NnetUpdater::nnet_, NnetUpdater::nnet_to_update_, CuMatrixBase< Real >::NumCols(), Nnet::NumComponents(), and CuMatrixBase< Real >::NumRows().

Referenced by NnetUpdater::ComputeForMinibatch().

                                                           {
  // We assume ComputeObjfAndDeriv has already been called.
  for (int32 c = nnet_.NumComponents() - 1;
       c >= nnet_.FirstUpdatableComponent(); c--) {
    const Component &component = nnet_.GetComponent(c);
    Component *component_to_update = (nnet_to_update_ == NULL ? NULL :
                                      &(nnet_to_update_->GetComponent(c)));
    const CuMatrix<BaseFloat> &input = forward_data_[c],
        &output = forward_data_[c+1];
    CuMatrix<BaseFloat> input_deriv(input.NumRows(), input.NumCols());
    const CuMatrix<BaseFloat> &output_deriv(*deriv);
    component.Backprop(chunk_info_out_[c], chunk_info_out_[c+1], input, output,                       
                       output_deriv, component_to_update,
                       &input_deriv);
    input_deriv.Swap(deriv);
  }
}

ComputeForMinibatch() [1/2]

double ComputeForMinibatch	(	const std::vector< NnetExample > &	data,
		double *	tot_accuracy
	)

Does the entire forward and backward computation for this minbatch.

Returns total objective function over this minibatch. If tot_accuracy != NULL, outputs to that pointer the total accuracy.

Definition at line 46 of file nnet-update.cc.

References NnetUpdater::Backprop(), NnetUpdater::ComputeObjfAndDeriv(), NnetUpdater::FormatInput(), NnetUpdater::nnet_to_update_, and NnetUpdater::Propagate().

Referenced by kaldi::nnet2::ComputeNnetObjf(), and kaldi::nnet2::DoBackprop().

                          {

  FormatInput(data);
  Propagate();
  CuMatrix<BaseFloat> tmp_deriv;
  double ans = ComputeObjfAndDeriv(data, &tmp_deriv, tot_accuracy);
  if (nnet_to_update_ != NULL)
    Backprop(&tmp_deriv); // this is summed (after weighting), not
                          // averaged.
  return ans;
}

ComputeForMinibatch() [2/2]

double ComputeForMinibatch	(	const std::vector< NnetExample > &	data,
		Matrix< BaseFloat > *	formatted_data,
		double *	tot_accuracy
	)

This version of ComputeForMinibatch is used when you have already called the function FormatNnetInput (defined below) to format your data as a single matrix.

This interface is provided because it can be more efficient to do this non-trivial CPU-based computation in a separate thread. formatted_data is an input but this function will destroy it, which is why it's a pointer.

Definition at line 63 of file nnet-update.cc.

References NnetUpdater::Backprop(), NnetUpdater::chunk_info_out_, Nnet::ComputeChunkInfo(), NnetUpdater::ComputeObjfAndDeriv(), NnetUpdater::forward_data_, Nnet::InputDim(), KALDI_ASSERT, Nnet::LeftContext(), NnetUpdater::nnet_, NnetUpdater::nnet_to_update_, MatrixBase< Real >::NumCols(), Nnet::NumComponents(), MatrixBase< Real >::NumRows(), NnetUpdater::Propagate(), and Nnet::RightContext().

                                                              {
  { // accept the formatted input.  This replaces the call to FormatInput().
    int32 num_chunks = data.size();
    KALDI_ASSERT(formatted_data->NumRows() ==
                 num_chunks * (1 + nnet_.LeftContext() + nnet_.RightContext()) &&
                 formatted_data->NumCols() == nnet_.InputDim());

    forward_data_.resize(nnet_.NumComponents() + 1);
    // the next command avoids the Swap() command ever copying GPU->CPU in case
    // an instance of this class is used more than once (which it isn't in
    // practice).
    forward_data_[0].Resize(0, 0);  
    forward_data_[0].Swap(formatted_data); // Copy to GPU, if being used.
    nnet_.ComputeChunkInfo(1 + nnet_.LeftContext() + nnet_.RightContext(),
                           data.size(), &chunk_info_out_);
  }
  Propagate();
  CuMatrix<BaseFloat> tmp_deriv;
  double ans = ComputeObjfAndDeriv(data, &tmp_deriv, tot_accuracy);
  if (nnet_to_update_ != NULL)
    Backprop(&tmp_deriv); // this is summed (after weighting), not
                          // averaged.
  return ans;
}

ComputeObjfAndDeriv()

double ComputeObjfAndDeriv ( const std::vector< NnetExample > &  data, CuMatrix< BaseFloat > *  deriv, double *  tot_accuracy = NULL  ) const

protected

Computes objective function and derivative at output layer, but does not do the backprop [for that, see Backprop()].

Returns objf summed over all samples (with their weights). If tot_accuracy != NULL, it will output to tot_accuracy the sum over all labels of all examples, of (correctly classified ? 0 : 1) * weight-of-label. This involves extra computation.

Definition at line 125 of file nnet-update.cc.

References CuMatrix< Real >::CompObjfAndDeriv(), NnetUpdater::ComputeTotAccuracy(), NnetUpdater::forward_data_, rnnlm::i, KALDI_ASSERT, KALDI_VLOG, NnetUpdater::nnet_, Nnet::NumComponents(), Nnet::OutputDim(), CuMatrix< Real >::Resize(), and kaldi::SameDim().

Referenced by NnetUpdater::ComputeForMinibatch().

                                {
  BaseFloat tot_objf = 0.0, tot_weight = 0.0;
  int32 num_components = nnet_.NumComponents();
  int32 num_chunks = data.size();
  deriv->Resize(num_chunks, nnet_.OutputDim()); // sets to zero.
  const CuMatrix<BaseFloat> &output(forward_data_[num_components]);
  KALDI_ASSERT(SameDim(output, *deriv));

  std::vector<MatrixElement<BaseFloat> > sv_labels;
  sv_labels.reserve(num_chunks); // We must have at least this many labels.
  for (int32 m = 0; m < num_chunks; m++) {
    KALDI_ASSERT(data[m].labels.size() == 1 &&
                 "Training code currently does not support multi-frame egs");
    const std::vector<std::pair<int32,BaseFloat> > &labels = data[m].labels[0];
    for (size_t i = 0; i < labels.size(); i++) {
      KALDI_ASSERT(labels[i].first < nnet_.OutputDim() &&
                        "Possibly egs come from alignments from mismatching model");
      MatrixElement<BaseFloat> elem = {m, labels[i].first, labels[i].second};
      sv_labels.push_back(elem);
    }
  }

  if (tot_accuracy != NULL)
    *tot_accuracy = ComputeTotAccuracy(data);
  
  deriv->CompObjfAndDeriv(sv_labels, output, &tot_objf, &tot_weight);
  
  KALDI_VLOG(4) << "Objective function is " << (tot_objf/tot_weight) << " over "
                << tot_weight << " samples (weighted).";
  return tot_objf;
}

ComputeTotAccuracy()

double ComputeTotAccuracy ( const std::vector< NnetExample > &  data ) const

private

Definition at line 161 of file nnet-update.cc.

References CuMatrixBase< Real >::FindRowMaxId(), NnetUpdater::forward_data_, rnnlm::i, rnnlm::j, KALDI_ASSERT, NnetUpdater::nnet_, Nnet::NumComponents(), and CuMatrixBase< Real >::NumRows().

Referenced by NnetUpdater::ComputeObjfAndDeriv().

                                              {
  BaseFloat tot_accuracy = 0.0;
  int32 num_components = nnet_.NumComponents();
  const CuMatrix<BaseFloat> &output(forward_data_[num_components]);
  KALDI_ASSERT(output.NumRows() == static_cast<int32>(data.size()));
  CuArray<int32> best_pdf(output.NumRows());
  std::vector<int32> best_pdf_cpu;
  
  output.FindRowMaxId(&best_pdf);
  best_pdf.CopyToVec(&best_pdf_cpu);

  for (int32 i = 0; i < output.NumRows(); i++) {
    KALDI_ASSERT(data[i].labels.size() == 1 &&
                 "Training code currently does not support multi-frame egs");
    const std::vector<std::pair<int32,BaseFloat> > &labels = data[i].labels[0];
    for (size_t j = 0; j < labels.size(); j++) {
      int32 ref_pdf_id = labels[j].first,
          hyp_pdf_id = best_pdf_cpu[i];
      BaseFloat weight = labels[j].second;
      tot_accuracy += weight * (hyp_pdf_id == ref_pdf_id ? 1.0 : 0.0);
    }
  }
  return tot_accuracy;
}

FormatInput()

void FormatInput ( const std::vector< NnetExample > &  data )

protected

Formats the input as a single matrix and sets the size of forward_data_, and sets up chunk_info_out_.

Definition at line 35 of file nnet-update.cc.

References NnetUpdater::chunk_info_out_, Nnet::ComputeChunkInfo(), kaldi::nnet2::FormatNnetInput(), NnetUpdater::forward_data_, Nnet::LeftContext(), NnetUpdater::nnet_, Nnet::NumComponents(), and Nnet::RightContext().

Referenced by NnetUpdater::ComputeForMinibatch().

                                                                {

  forward_data_.resize(nnet_.NumComponents() + 1);
  Matrix<BaseFloat> input;
  FormatNnetInput(nnet_, data, &input);
  forward_data_[0].Resize(0, 0);  // avoids the next command ever copying GPU->CPU
  forward_data_[0].Swap(&input); // Copy to GPU, if being used.
  nnet_.ComputeChunkInfo(1 + nnet_.LeftContext() + nnet_.RightContext(),
                         data.size(), &chunk_info_out_);
}

GetOutput()

void GetOutput

(

CuMatrix< BaseFloat > *

output

)

Definition at line 91 of file nnet-update.cc.

References NnetUpdater::forward_data_, KALDI_ASSERT, NnetUpdater::nnet_, and Nnet::NumComponents().

                                                       {
  int32 num_components = nnet_.NumComponents(); 
  KALDI_ASSERT(forward_data_.size() == nnet_.NumComponents() + 1); 
  *output = forward_data_[num_components];
}

Propagate()

void Propagate ( )

protected

Definition at line 97 of file nnet-update.cc.

References Component::BackpropNeedsInput(), Component::BackpropNeedsOutput(), NnetUpdater::chunk_info_out_, NnetUpdater::forward_data_, kaldi::g_kaldi_verbose_level, Nnet::GetComponent(), KALDI_VLOG, kaldi::kTrans, NnetUpdater::nnet_, Nnet::NumComponents(), Component::Propagate(), and kaldi::TraceMatMat().

Referenced by NnetUpdater::ComputeForMinibatch().

                            {
  static int32 num_times_printed = 0;
        
  int32 num_components = nnet_.NumComponents();
  for (int32 c = 0; c < num_components; c++) {
    const Component &component = nnet_.GetComponent(c);
    const CuMatrix<BaseFloat> &input = forward_data_[c];
    CuMatrix<BaseFloat> &output = forward_data_[c+1];
    // Note: the Propagate function will automatically resize the
    // output.
    component.Propagate(chunk_info_out_[c], chunk_info_out_[c+1], input, &output);
    // If we won't need the output of the previous layer for
    // backprop, delete it to save memory.
    bool need_last_output =
        (c>0 && nnet_.GetComponent(c-1).BackpropNeedsOutput()) ||
        component.BackpropNeedsInput();
    if (g_kaldi_verbose_level >= 3 && num_times_printed < 100) {
      KALDI_VLOG(3) << "Stddev of data for component " << c
                    << " for this minibatch is "
                    << (TraceMatMat(forward_data_[c], forward_data_[c], kTrans) /
                        (forward_data_[c].NumRows() * forward_data_[c].NumCols()));
      num_times_printed++;
    }
    if (!need_last_output)
      forward_data_[c].Resize(0, 0); // We won't need this data.
  }
}

Friends And Related Function Documentation

NnetEnsembleTrainer

friend class NnetEnsembleTrainer

friend

Definition at line 98 of file nnet-update.h.

Member Data Documentation

chunk_info_out_

std::vector<ChunkInfo> chunk_info_out_

private

Definition at line 106 of file nnet-update.h.

Referenced by NnetUpdater::Backprop(), NnetUpdater::ComputeForMinibatch(), NnetUpdater::FormatInput(), and NnetUpdater::Propagate().

forward_data_

std::vector<CuMatrix<BaseFloat> > forward_data_

private

Definition at line 108 of file nnet-update.h.

Referenced by NnetUpdater::Backprop(), NnetUpdater::ComputeForMinibatch(), NnetUpdater::ComputeObjfAndDeriv(), NnetUpdater::ComputeTotAccuracy(), NnetUpdater::FormatInput(), NnetUpdater::GetOutput(), and NnetUpdater::Propagate().

nnet_

const Nnet& nnet_

private

Definition at line 103 of file nnet-update.h.

Referenced by NnetUpdater::Backprop(), NnetUpdater::ComputeForMinibatch(), NnetUpdater::ComputeObjfAndDeriv(), NnetUpdater::ComputeTotAccuracy(), NnetUpdater::FormatInput(), NnetUpdater::GetOutput(), and NnetUpdater::Propagate().

nnet_to_update_

Nnet* nnet_to_update_

private

Definition at line 104 of file nnet-update.h.

Referenced by NnetUpdater::Backprop(), and NnetUpdater::ComputeForMinibatch().

num_chunks_

int32 num_chunks_

private

Definition at line 105 of file nnet-update.h.

---

The documentation for this class was generated from the following files:

• nnet2/nnet-update.h
• nnet2/nnet-update.cc