All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
NnetUpdater Class Reference

#include <nnet-update.h>

Collaboration diagram for NnetUpdater:

Public Member Functions

 NnetUpdater (const Nnet &nnet, Nnet *nnet_to_update)
 
double ComputeForMinibatch (const std::vector< NnetExample > &data, double *tot_accuracy)
 Does the entire forward and backward computation for this minbatch. More...
 
double ComputeForMinibatch (const std::vector< NnetExample > &data, Matrix< BaseFloat > *formatted_data, double *tot_accuracy)
 This version of ComputeForMinibatch is used when you have already called the function FormatNnetInput (defined below) to format your data as a single matrix. More...
 
void GetOutput (CuMatrix< BaseFloat > *output)
 

Protected Member Functions

void Propagate ()
 
void FormatInput (const std::vector< NnetExample > &data)
 Formats the input as a single matrix and sets the size of forward_data_, and sets up chunk_info_out_. More...
 
double ComputeObjfAndDeriv (const std::vector< NnetExample > &data, CuMatrix< BaseFloat > *deriv, double *tot_accuracy=NULL) const
 Computes objective function and derivative at output layer, but does not do the backprop [for that, see Backprop()]. More...
 
void Backprop (CuMatrix< BaseFloat > *deriv) const
 Backprop must be called after ComputeObjfAndDeriv. More...
 

Private Member Functions

double ComputeTotAccuracy (const std::vector< NnetExample > &data) const
 

Private Attributes

const Nnetnnet_
 
Nnetnnet_to_update_
 
int32 num_chunks_
 
std::vector< ChunkInfochunk_info_out_
 
std::vector< CuMatrix
< BaseFloat > > 
forward_data_
 

Friends

class NnetEnsembleTrainer
 

Detailed Description

Definition at line 46 of file nnet-update.h.

Constructor & Destructor Documentation

NnetUpdater ( const Nnet nnet,
Nnet nnet_to_update 
)

Definition at line 28 of file nnet-update.cc.

29  :
30  nnet_(nnet), nnet_to_update_(nnet_to_update) {
31 }

Member Function Documentation

void Backprop ( CuMatrix< BaseFloat > *  deriv) const
protected

Backprop must be called after ComputeObjfAndDeriv.

Does the backpropagation; "nnet_to_update_" is updated. Note: "deriv" will contain, at input, the derivative w.r.t. the output layer (as computed by ComputeObjfAndDeriv), but will be used as a temporary variable by this function.

Definition at line 188 of file nnet-update.cc.

References Component::Backprop(), NnetUpdater::chunk_info_out_, Nnet::FirstUpdatableComponent(), NnetUpdater::forward_data_, Nnet::GetComponent(), NnetUpdater::nnet_, NnetUpdater::nnet_to_update_, CuMatrixBase< Real >::NumCols(), Nnet::NumComponents(), and CuMatrixBase< Real >::NumRows().

Referenced by NnetUpdater::ComputeForMinibatch().

188  {
189  // We assume ComputeObjfAndDeriv has already been called.
190  for (int32 c = nnet_.NumComponents() - 1;
191  c >= nnet_.FirstUpdatableComponent(); c--) {
192  const Component &component = nnet_.GetComponent(c);
193  Component *component_to_update = (nnet_to_update_ == NULL ? NULL :
195  const CuMatrix<BaseFloat> &input = forward_data_[c],
196  &output = forward_data_[c+1];
197  CuMatrix<BaseFloat> input_deriv(input.NumRows(), input.NumCols());
198  const CuMatrix<BaseFloat> &output_deriv(*deriv);
199  component.Backprop(chunk_info_out_[c], chunk_info_out_[c+1], input, output,
200  output_deriv, component_to_update,
201  &input_deriv);
202  input_deriv.Swap(deriv);
203  }
204 }
int32 FirstUpdatableComponent() const
Returns the index of the lowest-numbered component which is updatable, or NumComponents() if none are...
Definition: nnet-nnet.cc:828
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
std::vector< ChunkInfo > chunk_info_out_
Definition: nnet-update.h:106
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
double ComputeForMinibatch ( const std::vector< NnetExample > &  data,
double *  tot_accuracy 
)

Does the entire forward and backward computation for this minbatch.

Returns total objective function over this minibatch. If tot_accuracy != NULL, outputs to that pointer the total accuracy.

Definition at line 46 of file nnet-update.cc.

References NnetUpdater::Backprop(), NnetUpdater::ComputeObjfAndDeriv(), NnetUpdater::FormatInput(), NnetUpdater::nnet_to_update_, and NnetUpdater::Propagate().

Referenced by kaldi::nnet2::ComputeNnetObjf(), and kaldi::nnet2::DoBackprop().

48  {
49 
50  FormatInput(data);
51  Propagate();
52  CuMatrix<BaseFloat> tmp_deriv;
53  double ans = ComputeObjfAndDeriv(data, &tmp_deriv, tot_accuracy);
54  if (nnet_to_update_ != NULL)
55  Backprop(&tmp_deriv); // this is summed (after weighting), not
56  // averaged.
57  return ans;
58 }
void FormatInput(const std::vector< NnetExample > &data)
Formats the input as a single matrix and sets the size of forward_data_, and sets up chunk_info_out_...
Definition: nnet-update.cc:35
double ComputeObjfAndDeriv(const std::vector< NnetExample > &data, CuMatrix< BaseFloat > *deriv, double *tot_accuracy=NULL) const
Computes objective function and derivative at output layer, but does not do the backprop [for that...
Definition: nnet-update.cc:125
void Backprop(CuMatrix< BaseFloat > *deriv) const
Backprop must be called after ComputeObjfAndDeriv.
Definition: nnet-update.cc:188
double ComputeForMinibatch ( const std::vector< NnetExample > &  data,
Matrix< BaseFloat > *  formatted_data,
double *  tot_accuracy 
)

This version of ComputeForMinibatch is used when you have already called the function FormatNnetInput (defined below) to format your data as a single matrix.

This interface is provided because it can be more efficient to do this non-trivial CPU-based computation in a separate thread. formatted_data is an input but this function will destroy it, which is why it's a pointer.

Definition at line 63 of file nnet-update.cc.

References NnetUpdater::Backprop(), NnetUpdater::chunk_info_out_, Nnet::ComputeChunkInfo(), NnetUpdater::ComputeObjfAndDeriv(), NnetUpdater::forward_data_, Nnet::InputDim(), KALDI_ASSERT, Nnet::LeftContext(), NnetUpdater::nnet_, NnetUpdater::nnet_to_update_, MatrixBase< Real >::NumCols(), Nnet::NumComponents(), MatrixBase< Real >::NumRows(), NnetUpdater::Propagate(), and Nnet::RightContext().

65  {
66  { // accept the formatted input. This replaces the call to FormatInput().
67  int32 num_chunks = data.size();
68  KALDI_ASSERT(formatted_data->NumRows() ==
69  num_chunks * (1 + nnet_.LeftContext() + nnet_.RightContext()) &&
70  formatted_data->NumCols() == nnet_.InputDim());
71 
72  forward_data_.resize(nnet_.NumComponents() + 1);
73  // the next command avoids the Swap() command ever copying GPU->CPU in case
74  // an instance of this class is used more than once (which it isn't in
75  // practice).
76  forward_data_[0].Resize(0, 0);
77  forward_data_[0].Swap(formatted_data); // Copy to GPU, if being used.
79  data.size(), &chunk_info_out_);
80  }
81  Propagate();
82  CuMatrix<BaseFloat> tmp_deriv;
83  double ans = ComputeObjfAndDeriv(data, &tmp_deriv, tot_accuracy);
84  if (nnet_to_update_ != NULL)
85  Backprop(&tmp_deriv); // this is summed (after weighting), not
86  // averaged.
87  return ans;
88 }
void ComputeChunkInfo(int32 input_chunk_size, int32 num_chunks, std::vector< ChunkInfo > *chunk_info_out) const
Uses the output of the Context() functions of the network, to compute a vector of size NumComponents(...
Definition: nnet-nnet.cc:65
int32 InputDim() const
Dimension of the input features, e.g.
Definition: nnet-nnet.cc:36
double ComputeObjfAndDeriv(const std::vector< NnetExample > &data, CuMatrix< BaseFloat > *deriv, double *tot_accuracy=NULL) const
Computes objective function and derivative at output layer, but does not do the backprop [for that...
Definition: nnet-update.cc:125
int32 LeftContext() const
Returns the left-context summed over all the Components...
Definition: nnet-nnet.cc:42
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
MatrixIndexT NumRows() const
Returns number of rows (or zero for emtpy matrix).
Definition: kaldi-matrix.h:58
MatrixIndexT NumCols() const
Returns number of columns (or zero for emtpy matrix).
Definition: kaldi-matrix.h:61
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Backprop(CuMatrix< BaseFloat > *deriv) const
Backprop must be called after ComputeObjfAndDeriv.
Definition: nnet-update.cc:188
std::vector< ChunkInfo > chunk_info_out_
Definition: nnet-update.h:106
int32 RightContext() const
Returns the right-context summed over all the Components...
Definition: nnet-nnet.cc:56
double ComputeObjfAndDeriv ( const std::vector< NnetExample > &  data,
CuMatrix< BaseFloat > *  deriv,
double *  tot_accuracy = NULL 
) const
protected

Computes objective function and derivative at output layer, but does not do the backprop [for that, see Backprop()].

Returns objf summed over all samples (with their weights). If tot_accuracy != NULL, it will output to tot_accuracy the sum over all labels of all examples, of (correctly classified ? 0 : 1) * weight-of-label. This involves extra computation.

Definition at line 125 of file nnet-update.cc.

References CuMatrix< Real >::CompObjfAndDeriv(), NnetUpdater::ComputeTotAccuracy(), NnetUpdater::forward_data_, rnnlm::i, KALDI_ASSERT, KALDI_VLOG, NnetUpdater::nnet_, Nnet::NumComponents(), Nnet::OutputDim(), CuMatrix< Real >::Resize(), and kaldi::SameDim().

Referenced by NnetUpdater::ComputeForMinibatch().

128  {
129  BaseFloat tot_objf = 0.0, tot_weight = 0.0;
130  int32 num_components = nnet_.NumComponents();
131  int32 num_chunks = data.size();
132  deriv->Resize(num_chunks, nnet_.OutputDim()); // sets to zero.
133  const CuMatrix<BaseFloat> &output(forward_data_[num_components]);
134  KALDI_ASSERT(SameDim(output, *deriv));
135 
136  std::vector<MatrixElement<BaseFloat> > sv_labels;
137  sv_labels.reserve(num_chunks); // We must have at least this many labels.
138  for (int32 m = 0; m < num_chunks; m++) {
139  KALDI_ASSERT(data[m].labels.size() == 1 &&
140  "Training code currently does not support multi-frame egs");
141  const std::vector<std::pair<int32,BaseFloat> > &labels = data[m].labels[0];
142  for (size_t i = 0; i < labels.size(); i++) {
143  KALDI_ASSERT(labels[i].first < nnet_.OutputDim() &&
144  "Possibly egs come from alignments from mismatching model");
145  MatrixElement<BaseFloat> elem = {m, labels[i].first, labels[i].second};
146  sv_labels.push_back(elem);
147  }
148  }
149 
150  if (tot_accuracy != NULL)
151  *tot_accuracy = ComputeTotAccuracy(data);
152 
153  deriv->CompObjfAndDeriv(sv_labels, output, &tot_objf, &tot_weight);
154 
155  KALDI_VLOG(4) << "Objective function is " << (tot_objf/tot_weight) << " over "
156  << tot_weight << " samples (weighted).";
157  return tot_objf;
158 }
void CompObjfAndDeriv(const std::vector< MatrixElement< Real > > &elements, const CuMatrix< Real > &A, Real *tot_objf, Real *tot_weight)
Here, A is interpreted as a matrix of probabilities, and "elements" as a list of posteriors (possibly...
Definition: cu-matrix.cc:1512
int32 OutputDim() const
The output dimension of the network – typically the number of pdfs.
Definition: nnet-nnet.cc:31
bool SameDim(const MatrixBase< Real > &M, const MatrixBase< Real > &N)
float BaseFloat
Definition: kaldi-types.h:29
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
double ComputeTotAccuracy(const std::vector< NnetExample > &data) const
Definition: nnet-update.cc:161
double ComputeTotAccuracy ( const std::vector< NnetExample > &  data) const
private

Definition at line 161 of file nnet-update.cc.

References CuMatrixBase< Real >::FindRowMaxId(), NnetUpdater::forward_data_, rnnlm::i, rnnlm::j, KALDI_ASSERT, NnetUpdater::nnet_, Nnet::NumComponents(), and CuMatrixBase< Real >::NumRows().

Referenced by NnetUpdater::ComputeObjfAndDeriv().

162  {
163  BaseFloat tot_accuracy = 0.0;
164  int32 num_components = nnet_.NumComponents();
165  const CuMatrix<BaseFloat> &output(forward_data_[num_components]);
166  KALDI_ASSERT(output.NumRows() == static_cast<int32>(data.size()));
167  CuArray<int32> best_pdf(output.NumRows());
168  std::vector<int32> best_pdf_cpu;
169 
170  output.FindRowMaxId(&best_pdf);
171  best_pdf.CopyToVec(&best_pdf_cpu);
172 
173  for (int32 i = 0; i < output.NumRows(); i++) {
174  KALDI_ASSERT(data[i].labels.size() == 1 &&
175  "Training code currently does not support multi-frame egs");
176  const std::vector<std::pair<int32,BaseFloat> > &labels = data[i].labels[0];
177  for (size_t j = 0; j < labels.size(); j++) {
178  int32 ref_pdf_id = labels[j].first,
179  hyp_pdf_id = best_pdf_cpu[i];
180  BaseFloat weight = labels[j].second;
181  tot_accuracy += weight * (hyp_pdf_id == ref_pdf_id ? 1.0 : 0.0);
182  }
183  }
184  return tot_accuracy;
185 }
float BaseFloat
Definition: kaldi-types.h:29
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void FormatInput ( const std::vector< NnetExample > &  data)
protected

Formats the input as a single matrix and sets the size of forward_data_, and sets up chunk_info_out_.

Definition at line 35 of file nnet-update.cc.

References NnetUpdater::chunk_info_out_, Nnet::ComputeChunkInfo(), kaldi::nnet2::FormatNnetInput(), NnetUpdater::forward_data_, Nnet::LeftContext(), NnetUpdater::nnet_, Nnet::NumComponents(), and Nnet::RightContext().

Referenced by NnetUpdater::ComputeForMinibatch().

35  {
36 
37  forward_data_.resize(nnet_.NumComponents() + 1);
38  Matrix<BaseFloat> input;
39  FormatNnetInput(nnet_, data, &input);
40  forward_data_[0].Resize(0, 0); // avoids the next command ever copying GPU->CPU
41  forward_data_[0].Swap(&input); // Copy to GPU, if being used.
43  data.size(), &chunk_info_out_);
44 }
void ComputeChunkInfo(int32 input_chunk_size, int32 num_chunks, std::vector< ChunkInfo > *chunk_info_out) const
Uses the output of the Context() functions of the network, to compute a vector of size NumComponents(...
Definition: nnet-nnet.cc:65
void FormatNnetInput(const Nnet &nnet, const std::vector< NnetExample > &data, Matrix< BaseFloat > *input_mat)
Takes the input to the nnet for a minibatch of examples, and formats as a single matrix.
Definition: nnet-update.cc:207
int32 LeftContext() const
Returns the left-context summed over all the Components...
Definition: nnet-nnet.cc:42
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
std::vector< ChunkInfo > chunk_info_out_
Definition: nnet-update.h:106
int32 RightContext() const
Returns the right-context summed over all the Components...
Definition: nnet-nnet.cc:56
void GetOutput ( CuMatrix< BaseFloat > *  output)

Definition at line 91 of file nnet-update.cc.

References NnetUpdater::forward_data_, KALDI_ASSERT, NnetUpdater::nnet_, and Nnet::NumComponents().

91  {
92  int32 num_components = nnet_.NumComponents();
94  *output = forward_data_[num_components];
95 }
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Propagate ( )
protected

Definition at line 97 of file nnet-update.cc.

References Component::BackpropNeedsInput(), Component::BackpropNeedsOutput(), NnetUpdater::chunk_info_out_, NnetUpdater::forward_data_, kaldi::g_kaldi_verbose_level, Nnet::GetComponent(), KALDI_VLOG, kaldi::kTrans, NnetUpdater::nnet_, Nnet::NumComponents(), Component::Propagate(), and kaldi::TraceMatMat().

Referenced by NnetUpdater::ComputeForMinibatch().

97  {
98  static int32 num_times_printed = 0;
99 
100  int32 num_components = nnet_.NumComponents();
101  for (int32 c = 0; c < num_components; c++) {
102  const Component &component = nnet_.GetComponent(c);
103  const CuMatrix<BaseFloat> &input = forward_data_[c];
104  CuMatrix<BaseFloat> &output = forward_data_[c+1];
105  // Note: the Propagate function will automatically resize the
106  // output.
107  component.Propagate(chunk_info_out_[c], chunk_info_out_[c+1], input, &output);
108  // If we won't need the output of the previous layer for
109  // backprop, delete it to save memory.
110  bool need_last_output =
111  (c>0 && nnet_.GetComponent(c-1).BackpropNeedsOutput()) ||
112  component.BackpropNeedsInput();
113  if (g_kaldi_verbose_level >= 3 && num_times_printed < 100) {
114  KALDI_VLOG(3) << "Stddev of data for component " << c
115  << " for this minibatch is "
117  (forward_data_[c].NumRows() * forward_data_[c].NumCols()));
118  num_times_printed++;
119  }
120  if (!need_last_output)
121  forward_data_[c].Resize(0, 0); // We won't need this data.
122  }
123 }
virtual bool BackpropNeedsOutput() const
std::vector< CuMatrix< BaseFloat > > forward_data_
Definition: nnet-update.h:108
Real TraceMatMat(const MatrixBase< Real > &A, const MatrixBase< Real > &B, MatrixTransposeType trans)
We need to declare this here as it will be a friend function.
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
int32 g_kaldi_verbose_level
This is set by util/parse-options.{h, cc} if you set –verbose=? option.
Definition: kaldi-error.cc:40
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
std::vector< ChunkInfo > chunk_info_out_
Definition: nnet-update.h:106
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141

Friends And Related Function Documentation

friend class NnetEnsembleTrainer
friend

Definition at line 98 of file nnet-update.h.

Member Data Documentation

std::vector<ChunkInfo> chunk_info_out_
private
Nnet* nnet_to_update_
private

Definition at line 104 of file nnet-update.h.

Referenced by NnetUpdater::Backprop(), and NnetUpdater::ComputeForMinibatch().

int32 num_chunks_
private

Definition at line 105 of file nnet-update.h.


The documentation for this class was generated from the following files: