All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
Xent Class Reference

#include <nnet-loss.h>

Inheritance diagram for Xent:
Collaboration diagram for Xent:

Public Member Functions

 Xent ()
 
 ~Xent ()
 
void Eval (const VectorBase< BaseFloat > &frame_weights, const CuMatrixBase< BaseFloat > &net_out, const CuMatrixBase< BaseFloat > &target, CuMatrix< BaseFloat > *diff)
 Evaluate cross entropy using target-matrix (supports soft labels),. More...
 
void Eval (const VectorBase< BaseFloat > &frame_weights, const CuMatrixBase< BaseFloat > &net_out, const Posterior &target, CuMatrix< BaseFloat > *diff)
 Evaluate cross entropy using target-posteriors (supports soft labels),. More...
 
std::string Report ()
 Generate string with error report,. More...
 
std::string ReportPerClass ()
 Generate string with per-class error report,. More...
 
BaseFloat AvgLoss ()
 Get loss value (frame average),. More...
 
- Public Member Functions inherited from LossItf
 LossItf ()
 
virtual ~LossItf ()
 

Private Attributes

CuVector< double > frames_
 
Vector< double > correct_
 
CuVector< double > xentropy_
 
CuVector< double > entropy_
 
double frames_progress_
 
double xentropy_progress_
 
double entropy_progress_
 
std::vector< float > loss_vec_
 
CuVector< BaseFloatframe_weights_
 
CuVector< BaseFloattarget_sum_
 
CuMatrix< BaseFloattgt_mat_
 
CuMatrix< BaseFloatframes_aux_
 
CuMatrix< BaseFloatxentropy_aux_
 
CuMatrix< BaseFloatentropy_aux_
 
CuArray< int32 > max_id_out_
 
CuArray< int32 > max_id_tgt_
 

Detailed Description

Definition at line 62 of file nnet-loss.h.

Constructor & Destructor Documentation

Xent ( )
inline

Definition at line 64 of file nnet-loss.h.

64  :
65  frames_progress_(0.0),
66  xentropy_progress_(0.0),
68  { }
double xentropy_progress_
Definition: nnet-loss.h:105
double entropy_progress_
Definition: nnet-loss.h:106
double frames_progress_
Definition: nnet-loss.h:104
~Xent ( )
inline

Definition at line 70 of file nnet-loss.h.

71  { }

Member Function Documentation

BaseFloat AvgLoss ( )
inlinevirtual

Get loss value (frame average),.

Implements LossItf.

Definition at line 92 of file nnet-loss.h.

References Xent::entropy_, Xent::frames_, CuVectorBase< Real >::Sum(), and Xent::xentropy_.

92  {
93  return (xentropy_.Sum() - entropy_.Sum()) / frames_.Sum();
94  }
CuVector< double > frames_
Definition: nnet-loss.h:98
CuVector< double > xentropy_
Definition: nnet-loss.h:100
CuVector< double > entropy_
Definition: nnet-loss.h:101
Real Sum() const
Definition: cu-vector.cc:260
void Eval ( const VectorBase< BaseFloat > &  frame_weights,
const CuMatrixBase< BaseFloat > &  net_out,
const CuMatrixBase< BaseFloat > &  target,
CuMatrix< BaseFloat > *  diff 
)
virtual

Evaluate cross entropy using target-matrix (supports soft labels),.

Implements LossItf.

Definition at line 61 of file nnet-loss.cc.

References CuMatrixBase< Real >::Add(), CuVectorBase< Real >::AddColSumMat(), CuMatrixBase< Real >::AddMat(), CuVectorBase< Real >::AddRowSumMat(), CuMatrixBase< Real >::ApplyLog(), Xent::correct_, kaldi::nnet1::CountCorrectFramesWeighted(), VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), Xent::entropy_, Xent::entropy_aux_, Xent::entropy_progress_, CuMatrixBase< Real >::FindRowMaxId(), Xent::frame_weights_, Xent::frames_, Xent::frames_aux_, Xent::frames_progress_, KALDI_ASSERT, KALDI_ISFINITE, KALDI_VLOG, Xent::loss_vec_, Xent::max_id_out_, Xent::max_id_tgt_, CuVectorBase< Real >::MulElements(), CuMatrixBase< Real >::MulElements(), CuMatrixBase< Real >::MulRowsVec(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), CuVector< Real >::Resize(), Vector< Real >::Resize(), CuVectorBase< Real >::Sum(), VectorBase< Real >::Sum(), CuMatrixBase< Real >::Sum(), Xent::target_sum_, Xent::xentropy_, Xent::xentropy_aux_, and Xent::xentropy_progress_.

Referenced by Xent::Eval(), and main().

64  {
65  // check inputs,
66  KALDI_ASSERT(net_out.NumCols() == targets.NumCols());
67  KALDI_ASSERT(net_out.NumRows() == targets.NumRows());
68  KALDI_ASSERT(net_out.NumRows() == frame_weights.Dim());
69 
70  KALDI_ASSERT(KALDI_ISFINITE(frame_weights.Sum()));
71  KALDI_ASSERT(KALDI_ISFINITE(net_out.Sum()));
72  KALDI_ASSERT(KALDI_ISFINITE(targets.Sum()));
73 
74  // buffer initialization,
75  int32 num_classes = targets.NumCols();
76  if (frames_.Dim() == 0) {
77  frames_.Resize(num_classes);
78  xentropy_.Resize(num_classes);
79  entropy_.Resize(num_classes);
80  correct_.Resize(num_classes);
81  }
82 
83  // get frame_weights to GPU,
84  frame_weights_ = frame_weights;
85 
86  // There may be frames for which the sum of targets is zero.
87  // This happens in multi-lingual training when the frame
88  // has target class in the softmax of another language.
89  // We 'switch-off' such frames by masking the 'frame_weights_',
90  target_sum_.Resize(targets.NumRows());
91  target_sum_.AddColSumMat(1.0, targets, 0.0);
93 
94  // compute derivative wrt. activations of last layer of neurons,
95  *diff = net_out;
96  diff->AddMat(-1.0, targets);
97  diff->MulRowsVec(frame_weights_); // weighting,
98 
99  // count frames per class,
100  frames_aux_ = targets;
102  frames_.AddRowSumMat(1.0, CuMatrix<double>(frames_aux_));
103 
104  // evaluate the frame-level classification,
105  net_out.FindRowMaxId(&max_id_out_); // find max in nn-output
106  targets.FindRowMaxId(&max_id_tgt_); // find max in targets
109 
110  // calculate cross_entropy (in GPU),
111  xentropy_aux_ = net_out; // y
112  xentropy_aux_.Add(1e-20); // avoid log(0)
113  xentropy_aux_.ApplyLog(); // log(y)
114  xentropy_aux_.MulElements(targets); // t*log(y)
115  xentropy_aux_.MulRowsVec(frame_weights_); // w*t*log(y)
116  xentropy_.AddRowSumMat(-1.0, CuMatrix<double>(xentropy_aux_));
117 
118  // caluculate entropy (in GPU),
119  entropy_aux_ = targets; // t
120  entropy_aux_.Add(1e-20); // avoid log(0)
121  entropy_aux_.ApplyLog(); // log(t)
122  entropy_aux_.MulElements(targets); // t*log(t)
123  entropy_aux_.MulRowsVec(frame_weights_); // w*t*log(t)
124  entropy_.AddRowSumMat(-1.0, CuMatrix<double>(entropy_aux_));
125 
126  // progressive loss reporting
127  {
128  static const int32 progress_step = 3600*100; // 1h
132 
135 
136  if (frames_progress_ > progress_step) {
137  double progress_value =
139  // print,
140  KALDI_VLOG(1) << "ProgressLoss[last "
141  << static_cast<int>(frames_progress_/100/3600) << "h of "
142  << static_cast<int>(frames_.Sum()/100/3600) << "h]: "
143  << progress_value << " (Xent)";
144  // store,
145  loss_vec_.push_back(progress_value);
146  // reset,
147  frames_progress_ = 0;
148  xentropy_progress_ = 0.0;
149  entropy_progress_ = 0.0;
150  }
151  }
152 }
double xentropy_progress_
Definition: nnet-loss.h:105
void FindRowMaxId(CuArray< int32 > *id) const
Find the id of the maximal element for each row.
Definition: cu-matrix.cc:1611
CuVector< double > frames_
Definition: nnet-loss.h:98
CuMatrix< BaseFloat > frames_aux_
Definition: nnet-loss.h:115
Real Sum() const
Definition: cu-matrix.cc:2621
CuVector< BaseFloat > target_sum_
Definition: nnet-loss.h:111
void AddColSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the columns of the matrix, add to vector.
Definition: cu-vector.cc:1170
#define KALDI_ISFINITE(x)
Definition: kaldi-math.h:74
std::vector< float > loss_vec_
Definition: nnet-loss.h:107
CuVector< double > xentropy_
Definition: nnet-loss.h:100
Real Sum() const
Returns sum of the elements.
void MulElements(const CuVectorBase< Real > &v)
Definition: cu-vector.cc:743
MatrixIndexT NumCols() const
Definition: cu-matrix.h:201
void Resize(MatrixIndexT length, MatrixResizeType resize_type=kSetZero)
Set vector to a specified size (can be zero).
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:883
CuMatrix< BaseFloat > entropy_aux_
Definition: nnet-loss.h:117
void CountCorrectFramesWeighted(const CuArray< T > &hyp, const CuArray< T > &ref, const CuVectorBase< BaseFloat > &weights, Vector< double > *correct)
Helper function of Xent::Eval, calculates number of matching elemente in 'hyp', 'ref' weighted by 'we...
Definition: nnet-loss.cc:40
CuArray< int32 > max_id_out_
Definition: nnet-loss.h:120
Vector< double > correct_
Definition: nnet-loss.h:99
void MulRowsVec(const CuVectorBase< Real > &scale)
scale i'th row by scale[i]
Definition: cu-matrix.cc:752
double entropy_progress_
Definition: nnet-loss.h:106
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1157
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:200
CuVector< double > entropy_
Definition: nnet-loss.h:101
CuVector< BaseFloat > frame_weights_
Definition: nnet-loss.h:110
CuMatrix< BaseFloat > xentropy_aux_
Definition: nnet-loss.h:116
void AddMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType trans=kNoTrans)
*this += alpha * A
Definition: cu-matrix.cc:913
CuArray< int32 > max_id_tgt_
Definition: nnet-loss.h:121
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void MulElements(const CuMatrixBase< Real > &A)
Multiply two matrices elementwise: C = C .* A.
Definition: cu-matrix.cc:652
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
double frames_progress_
Definition: nnet-loss.h:104
void Add(Real value)
Definition: cu-matrix.cc:546
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:59
Real Sum() const
Definition: cu-vector.cc:260
void Eval ( const VectorBase< BaseFloat > &  frame_weights,
const CuMatrixBase< BaseFloat > &  net_out,
const Posterior target,
CuMatrix< BaseFloat > *  diff 
)
virtual

Evaluate cross entropy using target-posteriors (supports soft labels),.

Implements LossItf.

Definition at line 155 of file nnet-loss.cc.

References Xent::Eval(), KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), kaldi::nnet1::PosteriorToMatrix(), and Xent::tgt_mat_.

158  {
159  int32 num_frames = net_out.NumRows(),
160  num_pdf = net_out.NumCols();
161  KALDI_ASSERT(num_frames == post.size());
162 
163  // convert posterior to matrix,
164  PosteriorToMatrix(post, num_pdf, &tgt_mat_);
165 
166  // call the other eval function,
167  Eval(frame_weights, net_out, tgt_mat_, diff);
168 }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:201
void PosteriorToMatrix(const Posterior &post, const int32 post_dim, CuMatrix< Real > *mat)
Wrapper of PosteriorToMatrix with CuMatrix argument.
Definition: nnet-utils.h:292
CuMatrix< BaseFloat > tgt_mat_
Definition: nnet-loss.h:114
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:200
void Eval(const VectorBase< BaseFloat > &frame_weights, const CuMatrixBase< BaseFloat > &net_out, const CuMatrixBase< BaseFloat > &target, CuMatrix< BaseFloat > *diff)
Evaluate cross entropy using target-matrix (supports soft labels),.
Definition: nnet-loss.cc:61
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
std::string Report ( )
virtual

Generate string with error report,.

Implements LossItf.

Definition at line 171 of file nnet-loss.cc.

References Xent::correct_, Xent::entropy_, Xent::frames_, Xent::loss_vec_, CuVectorBase< Real >::Sum(), VectorBase< Real >::Sum(), and Xent::xentropy_.

Referenced by main().

171  {
172  double loss_value =
173  (xentropy_.Sum() - entropy_.Sum()) / frames_.Sum();
174  std::ostringstream oss;
175  oss << "AvgLoss: " << loss_value << " (Xent), "
176  << "[AvgXent: " << xentropy_.Sum() / frames_.Sum()
177  << ", AvgTargetEnt: " << entropy_.Sum() / frames_.Sum()
178  << "]" << std::endl;
179 
180  oss << "progress: [";
181  std::copy(loss_vec_.begin(), loss_vec_.end(),
182  std::ostream_iterator<float>(oss, " "));
183  oss << "]" << std::endl;
184 
185  double frame_accuracy = 100.0 * correct_.Sum() / frames_.Sum();
186  oss << "FRAME_ACCURACY >> " << frame_accuracy << "% <<" << std::endl;
187 
188  return oss.str();
189 }
CuVector< double > frames_
Definition: nnet-loss.h:98
std::vector< float > loss_vec_
Definition: nnet-loss.h:107
CuVector< double > xentropy_
Definition: nnet-loss.h:100
Real Sum() const
Returns sum of the elements.
Vector< double > correct_
Definition: nnet-loss.h:99
CuVector< double > entropy_
Definition: nnet-loss.h:101
Real Sum() const
Definition: cu-vector.cc:260
std::string ReportPerClass ( )

Generate string with per-class error report,.

Definition at line 192 of file nnet-loss.cc.

References CuVectorBase< Real >::Add(), CuVectorBase< Real >::AddVec(), CuVectorBase< Real >::ApplyPow(), Xent::correct_, Xent::entropy_, Xent::frames_, CuVectorBase< Real >::MulElements(), CuVectorBase< Real >::Scale(), and Xent::xentropy_.

Referenced by main().

192  {
193  std::ostringstream oss;
194  oss << "PER-CLASS PERFORMANCE:" << std::endl;
195  oss << "@@@ Frames per-class:" << frames_;
196  // get inverted counts,
197  CuVector<double> inv_frames(frames_);
198  inv_frames.Add(0.5); // avoid 0-frames,
199  inv_frames.ApplyPow(-1.0);
200  // loss, kl = xentropy-entropy,
201  CuVector<double> loss(xentropy_);
202  loss.AddVec(-1.0, entropy_);
203  loss.MulElements(inv_frames);
204  oss << "@@@ Loss per-class:" << loss;
205  // frame accuracy (assuming targets are binary),
206  CuVector<double> frm_accu(correct_);
207  frm_accu.MulElements(inv_frames);
208  frm_accu.Scale(100.0);
209  oss << "@@@ Frame-accuracy per-class:" << frm_accu;
210  //
211  return oss.str();
212 }
CuVector< double > frames_
Definition: nnet-loss.h:98
CuVector< double > xentropy_
Definition: nnet-loss.h:100
Vector< double > correct_
Definition: nnet-loss.h:99
CuVector< double > entropy_
Definition: nnet-loss.h:101

Member Data Documentation

Vector<double> correct_
private

Definition at line 99 of file nnet-loss.h.

Referenced by Xent::Eval(), Xent::Report(), and Xent::ReportPerClass().

CuVector<double> entropy_
private

Definition at line 101 of file nnet-loss.h.

Referenced by Xent::AvgLoss(), Xent::Eval(), Xent::Report(), and Xent::ReportPerClass().

CuMatrix<BaseFloat> entropy_aux_
private

Definition at line 117 of file nnet-loss.h.

Referenced by Xent::Eval().

double entropy_progress_
private

Definition at line 106 of file nnet-loss.h.

Referenced by Xent::Eval().

CuVector<BaseFloat> frame_weights_
private

Definition at line 110 of file nnet-loss.h.

Referenced by Xent::Eval().

CuVector<double> frames_
private

Definition at line 98 of file nnet-loss.h.

Referenced by Xent::AvgLoss(), Xent::Eval(), Xent::Report(), and Xent::ReportPerClass().

CuMatrix<BaseFloat> frames_aux_
private

Definition at line 115 of file nnet-loss.h.

Referenced by Xent::Eval().

double frames_progress_
private

Definition at line 104 of file nnet-loss.h.

Referenced by Xent::Eval().

std::vector<float> loss_vec_
private

Definition at line 107 of file nnet-loss.h.

Referenced by Xent::Eval(), and Xent::Report().

CuArray<int32> max_id_out_
private

Definition at line 120 of file nnet-loss.h.

Referenced by Xent::Eval().

CuArray<int32> max_id_tgt_
private

Definition at line 121 of file nnet-loss.h.

Referenced by Xent::Eval().

CuVector<BaseFloat> target_sum_
private

Definition at line 111 of file nnet-loss.h.

Referenced by Xent::Eval().

CuMatrix<BaseFloat> tgt_mat_
private

Definition at line 114 of file nnet-loss.h.

Referenced by Xent::Eval().

CuVector<double> xentropy_
private

Definition at line 100 of file nnet-loss.h.

Referenced by Xent::AvgLoss(), Xent::Eval(), Xent::Report(), and Xent::ReportPerClass().

CuMatrix<BaseFloat> xentropy_aux_
private

Definition at line 116 of file nnet-loss.h.

Referenced by Xent::Eval().

double xentropy_progress_
private

Definition at line 105 of file nnet-loss.h.

Referenced by Xent::Eval().


The documentation for this class was generated from the following files: