All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
LstmProjected Class Reference

#include <nnet-lstm-projected.h>

Inheritance diagram for LstmProjected:
Collaboration diagram for LstmProjected:

Public Member Functions

 LstmProjected (int32 input_dim, int32 output_dim)
 
 ~LstmProjected ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void InitData (std::istream &is)
 Initialize the content of the component by the 'line' from the prototype,. More...
 
void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
int32 NumParams () const
 Number of trainable parameters,. More...
 
void GetGradient (VectorBase< BaseFloat > *gradient) const
 Get gradient reshaped as a vector,. More...
 
void GetParams (VectorBase< BaseFloat > *params) const
 Get the trainable parameters reshaped as a vector,. More...
 
void SetParams (const VectorBase< BaseFloat > &params)
 Set the trainable parameters from, reshaped as a vector,. More...
 
std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 
void ResetStreams (const std::vector< int32 > &stream_reset_flag)
 TODO: Do we really need this? More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
void Update (const CuMatrixBase< BaseFloat > &input, const CuMatrixBase< BaseFloat > &diff)
 Compute gradient and update parameters,. More...
 
- Public Member Functions inherited from MultistreamComponent
 MultistreamComponent (int32 input_dim, int32 output_dim)
 
bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
virtual void SetSeqLengths (const std::vector< int32 > &sequence_lengths)
 
int32 NumStreams () const
 
- Public Member Functions inherited from UpdatableComponent
 UpdatableComponent (int32 input_dim, int32 output_dim)
 
virtual ~UpdatableComponent ()
 
bool IsUpdatable () const
 Check if contains trainable parameters,. More...
 
virtual void SetTrainOptions (const NnetTrainOptions &opts)
 Set the training options to the component,. More...
 
const NnetTrainOptionsGetTrainOptions () const
 Get the training options from the component,. More...
 
virtual void SetLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient,. More...
 
virtual void SetBiasLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient for bias,. More...
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 

Private Attributes

int32 cell_dim_
 
int32 proj_dim_
 recurrent projection layer dim More...
 
BaseFloat cell_clip_
 Clipping of 'cell-values' in forward pass (per-frame),. More...
 
BaseFloat diff_clip_
 Clipping of 'derivatives' in backprop (per-frame),. More...
 
BaseFloat cell_diff_clip_
 Clipping of 'cell-derivatives' accumulated over CEC (per-frame),. More...
 
BaseFloat grad_clip_
 Clipping of the updates,. More...
 
CuMatrix< BaseFloatprev_nnet_state_
 
BaseFloat clip_gradient_
 
CuMatrix< BaseFloatw_gifo_x_
 
CuMatrix< BaseFloatw_gifo_x_corr_
 
CuMatrix< BaseFloatw_gifo_r_
 
CuMatrix< BaseFloatw_gifo_r_corr_
 
CuVector< BaseFloatbias_
 
CuVector< BaseFloatbias_corr_
 
CuVector< BaseFloatpeephole_i_c_
 
CuVector< BaseFloatpeephole_f_c_
 
CuVector< BaseFloatpeephole_o_c_
 
CuVector< BaseFloatpeephole_i_c_corr_
 
CuVector< BaseFloatpeephole_f_c_corr_
 
CuVector< BaseFloatpeephole_o_c_corr_
 
CuMatrix< BaseFloatw_r_m_
 
CuMatrix< BaseFloatw_r_m_corr_
 
CuMatrix< BaseFloatpropagate_buf_
 
CuMatrix< BaseFloatbackpropagate_buf_
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 
- Protected Attributes inherited from MultistreamComponent
std::vector< int32 > sequence_lengths_
 
- Protected Attributes inherited from UpdatableComponent
NnetTrainOptions opts_
 Option-class with training hyper-parameters,. More...
 
BaseFloat learn_rate_coef_
 Scalar applied to learning rate for weight matrices (to be used in ::Update method),. More...
 
BaseFloat bias_learn_rate_coef_
 Scalar applied to learning rate for bias (to be used in ::Update method),. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Detailed Description

Definition at line 48 of file nnet-lstm-projected.h.

Constructor & Destructor Documentation

LstmProjected ( int32  input_dim,
int32  output_dim 
)
inline

Definition at line 50 of file nnet-lstm-projected.h.

Referenced by LstmProjected::Copy().

50  :
51  MultistreamComponent(input_dim, output_dim),
52  cell_dim_(0),
53  proj_dim_(output_dim),
54  cell_clip_(50.0),
55  diff_clip_(1.0),
56  cell_diff_clip_(0.0),
57  grad_clip_(250.0)
58  { }
BaseFloat cell_diff_clip_
Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
MultistreamComponent(int32 input_dim, int32 output_dim)
BaseFloat grad_clip_
Clipping of the updates,.
int32 proj_dim_
recurrent projection layer dim
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
~LstmProjected ( )
inline

Definition at line 60 of file nnet-lstm-projected.h.

61  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 475 of file nnet-lstm-projected.h.

References CuVectorBase< Real >::AddDiagMatMat(), CuMatrixBase< Real >::AddMatMat(), CuVectorBase< Real >::AddRowSumMat(), LstmProjected::backpropagate_buf_, LstmProjected::bias_corr_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, CuMatrixBase< Real >::ColRange(), LstmProjected::diff_clip_, Component::input_dim_, kaldi::kNoTrans, kaldi::kSetZero, kaldi::kTrans, NnetTrainOptions::momentum, CuMatrixBase< Real >::NumRows(), MultistreamComponent::NumStreams(), UpdatableComponent::opts_, LstmProjected::peephole_f_c_, LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_, LstmProjected::peephole_o_c_corr_, LstmProjected::proj_dim_, LstmProjected::propagate_buf_, CuVector< Real >::Resize(), CuMatrix< Real >::Resize(), CuMatrixBase< Real >::RowRange(), MultistreamComponent::sequence_lengths_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_, LstmProjected::w_gifo_x_corr_, LstmProjected::w_r_m_, and LstmProjected::w_r_m_corr_.

478  {
479 
480  // the number of sequences to be processed in parallel
481  int32 T = in.NumRows() / NumStreams();
482  int32 S = NumStreams();
483 
484  // buffer,
486 
487  // split activations by neuron types,
488  CuSubMatrix<BaseFloat> YG(propagate_buf_.ColRange(0*cell_dim_, cell_dim_));
489  CuSubMatrix<BaseFloat> YI(propagate_buf_.ColRange(1*cell_dim_, cell_dim_));
490  CuSubMatrix<BaseFloat> YF(propagate_buf_.ColRange(2*cell_dim_, cell_dim_));
491  CuSubMatrix<BaseFloat> YO(propagate_buf_.ColRange(3*cell_dim_, cell_dim_));
492  CuSubMatrix<BaseFloat> YC(propagate_buf_.ColRange(4*cell_dim_, cell_dim_));
493  CuSubMatrix<BaseFloat> YH(propagate_buf_.ColRange(5*cell_dim_, cell_dim_));
494  CuSubMatrix<BaseFloat> YM(propagate_buf_.ColRange(6*cell_dim_, cell_dim_));
495  CuSubMatrix<BaseFloat> YR(propagate_buf_.ColRange(7*cell_dim_, proj_dim_));
496 
497  // split derivatives by neuron types,
498  CuSubMatrix<BaseFloat> DG(backpropagate_buf_.ColRange(0*cell_dim_, cell_dim_));
499  CuSubMatrix<BaseFloat> DI(backpropagate_buf_.ColRange(1*cell_dim_, cell_dim_));
500  CuSubMatrix<BaseFloat> DF(backpropagate_buf_.ColRange(2*cell_dim_, cell_dim_));
501  CuSubMatrix<BaseFloat> DO(backpropagate_buf_.ColRange(3*cell_dim_, cell_dim_));
502  CuSubMatrix<BaseFloat> DC(backpropagate_buf_.ColRange(4*cell_dim_, cell_dim_));
503  CuSubMatrix<BaseFloat> DH(backpropagate_buf_.ColRange(5*cell_dim_, cell_dim_));
504  CuSubMatrix<BaseFloat> DM(backpropagate_buf_.ColRange(6*cell_dim_, cell_dim_));
505  CuSubMatrix<BaseFloat> DR(backpropagate_buf_.ColRange(7*cell_dim_, proj_dim_));
506  CuSubMatrix<BaseFloat> DGIFO(backpropagate_buf_.ColRange(0, 4*cell_dim_));
507 
508  // pre-copy partial derivatives from the LSTM output,
509  DR.RowRange(1*S, T*S).CopyFromMat(out_diff);
510 
511  // BufferPadding [T0]:dummy, [1,T]:current sequence, [T+1]: dummy,
512  for (int t = T; t >= 1; t--) {
513  CuSubMatrix<BaseFloat> y_g(YG.RowRange(t*S, S));
514  CuSubMatrix<BaseFloat> y_i(YI.RowRange(t*S, S));
515  CuSubMatrix<BaseFloat> y_f(YF.RowRange(t*S, S));
516  CuSubMatrix<BaseFloat> y_o(YO.RowRange(t*S, S));
517  CuSubMatrix<BaseFloat> y_c(YC.RowRange(t*S, S));
518  CuSubMatrix<BaseFloat> y_h(YH.RowRange(t*S, S));
519  CuSubMatrix<BaseFloat> y_m(YM.RowRange(t*S, S));
520  CuSubMatrix<BaseFloat> y_r(YR.RowRange(t*S, S));
521 
522  CuSubMatrix<BaseFloat> d_all(backpropagate_buf_.RowRange(t*S, S));
523  CuSubMatrix<BaseFloat> d_g(DG.RowRange(t*S, S));
524  CuSubMatrix<BaseFloat> d_i(DI.RowRange(t*S, S));
525  CuSubMatrix<BaseFloat> d_f(DF.RowRange(t*S, S));
526  CuSubMatrix<BaseFloat> d_o(DO.RowRange(t*S, S));
527  CuSubMatrix<BaseFloat> d_c(DC.RowRange(t*S, S));
528  CuSubMatrix<BaseFloat> d_h(DH.RowRange(t*S, S));
529  CuSubMatrix<BaseFloat> d_m(DM.RowRange(t*S, S));
530  CuSubMatrix<BaseFloat> d_r(DR.RowRange(t*S, S));
531  CuSubMatrix<BaseFloat> d_gifo(DGIFO.RowRange(t*S, S));
532 
533  // r
534  // Version 1 (precise gradients):
535  // backprop error from g(t+1), i(t+1), f(t+1), o(t+1) to r(t)
536  d_r.AddMatMat(1.0, DGIFO.RowRange((t+1)*S, S), kNoTrans, w_gifo_r_, kNoTrans, 1.0);
537 
538  /*
539  // Version 2 (Alex Graves' PhD dissertation):
540  // only backprop g(t+1) to r(t)
541  CuSubMatrix<BaseFloat> w_g_r_(w_gifo_r_.RowRange(0, cell_dim_));
542  d_r.AddMatMat(1.0, DG.RowRange((t+1)*S,S), kNoTrans, w_g_r_, kNoTrans, 1.0);
543  */
544 
545  /*
546  // Version 3 (Felix Gers' PhD dissertation):
547  // truncate gradients of g(t+1), i(t+1), f(t+1), o(t+1) once they leak out memory block
548  // CEC(with forget connection) is the only "error-bridge" through time
549  */
550 
551  // r -> m
552  d_m.AddMatMat(1.0, d_r, kNoTrans, w_r_m_, kNoTrans, 0.0);
553 
554  // m -> h via output gate
555  d_h.AddMatMatElements(1.0, d_m, y_o, 0.0);
556  d_h.DiffTanh(y_h, d_h);
557 
558  // o
559  d_o.AddMatMatElements(1.0, d_m, y_h, 0.0);
560  d_o.DiffSigmoid(y_o, d_o);
561 
562  // c
563  // 1. diff from h(t)
564  // 2. diff from c(t+1) (via forget-gate between CEC)
565  // 3. diff from i(t+1) (via peephole)
566  // 4. diff from f(t+1) (via peephole)
567  // 5. diff from o(t) (via peephole, not recurrent)
568  d_c.AddMat(1.0, d_h);
569  d_c.AddMatMatElements(1.0, DC.RowRange((t+1)*S, S), YF.RowRange((t+1)*S,S), 1.0);
570  d_c.AddMatDiagVec(1.0, DI.RowRange((t+1)*S, S), kNoTrans, peephole_i_c_, 1.0);
571  d_c.AddMatDiagVec(1.0, DF.RowRange((t+1)*S, S), kNoTrans, peephole_f_c_, 1.0);
572  d_c.AddMatDiagVec(1.0, d_o , kNoTrans, peephole_o_c_, 1.0);
573  // optionally clip the cell_derivative,
574  if (cell_diff_clip_ > 0.0) {
575  d_c.ApplyFloor(-cell_diff_clip_);
576  d_c.ApplyCeiling(cell_diff_clip_);
577  }
578 
579  // f
580  d_f.AddMatMatElements(1.0, d_c, YC.RowRange((t-1)*S,S), 0.0);
581  d_f.DiffSigmoid(y_f, d_f);
582 
583  // i
584  d_i.AddMatMatElements(1.0, d_c, y_g, 0.0);
585  d_i.DiffSigmoid(y_i, d_i);
586 
587  // c -> g via input gate
588  d_g.AddMatMatElements(1.0, d_c, y_i, 0.0);
589  d_g.DiffTanh(y_g, d_g);
590 
591  // Clipping per-frame derivatives for the next `t'.
592  // Clipping applied to gates and input gate (as done in Google).
593  // [ICASSP2015, Sak, Learning acoustic frame labelling...],
594  //
595  // The path from 'out_diff' to 'd_c' via 'd_h' is unclipped,
596  // which is probably important for the 'Constant Error Carousel'
597  // to work well.
598  //
599  if (diff_clip_ > 0.0) {
600  d_gifo.ApplyFloor(-diff_clip_);
601  d_gifo.ApplyCeiling(diff_clip_);
602  }
603 
604  // set zeros to padded frames,
605  if (sequence_lengths_.size() > 0) {
606  for (int s = 0; s < S; s++) {
607  if (t > sequence_lengths_[s]) {
608  d_all.Row(s).SetZero();
609  }
610  }
611  }
612  }
613 
614  // g,i,f,o -> x, calculating input derivatives,
615  in_diff->AddMatMat(1.0, DGIFO.RowRange(1*S,T*S), kNoTrans, w_gifo_x_, kNoTrans, 0.0);
616 
617  // lazy initialization of udpate buffers,
618  if (w_gifo_x_corr_.NumRows() == 0) {
626  }
627 
628  // calculate delta
629  const BaseFloat mmt = opts_.momentum;
630 
631  // weight x -> g, i, f, o
632  w_gifo_x_corr_.AddMatMat(1.0, DGIFO.RowRange(1*S, T*S), kTrans,
633  in , kNoTrans, mmt);
634  // recurrent weight r -> g, i, f, o
635  w_gifo_r_corr_.AddMatMat(1.0, DGIFO.RowRange(1*S, T*S), kTrans,
636  YR.RowRange(0*S, T*S) , kNoTrans, mmt);
637  // bias of g, i, f, o
638  bias_corr_.AddRowSumMat(1.0, DGIFO.RowRange(1*S, T*S), mmt);
639 
640  // recurrent peephole c -> i
641  peephole_i_c_corr_.AddDiagMatMat(1.0, DI.RowRange(1*S, T*S), kTrans,
642  YC.RowRange(0*S, T*S), kNoTrans, mmt);
643  // recurrent peephole c -> f
644  peephole_f_c_corr_.AddDiagMatMat(1.0, DF.RowRange(1*S, T*S), kTrans,
645  YC.RowRange(0*S, T*S), kNoTrans, mmt);
646  // peephole c -> o
647  peephole_o_c_corr_.AddDiagMatMat(1.0, DO.RowRange(1*S, T*S), kTrans,
648  YC.RowRange(1*S, T*S), kNoTrans, mmt);
649 
650  w_r_m_corr_.AddMatMat(1.0, DR.RowRange(1*S, T*S), kTrans,
651  YM.RowRange(1*S, T*S), kNoTrans, mmt);
652  }
BaseFloat cell_diff_clip_
Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.
CuMatrix< BaseFloat > backpropagate_buf_
CuVector< BaseFloat > peephole_f_c_corr_
NnetTrainOptions opts_
Option-class with training hyper-parameters,.
int32 input_dim_
Data members,.
CuMatrix< BaseFloat > propagate_buf_
CuMatrix< BaseFloat > w_gifo_r_corr_
CuVector< BaseFloat > peephole_o_c_
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
CuSubMatrix< Real > RowRange(const MatrixIndexT row_offset, const MatrixIndexT num_rows) const
Definition: cu-matrix.h:539
void AddDiagMatMat(Real alpha, const CuMatrixBase< Real > &M, MatrixTransposeType transM, const CuMatrixBase< Real > &N, MatrixTransposeType transN, Real beta=1.0)
Add the diagonal of a matrix product: *this = diag(M N), assuming the "trans" arguments are both kNoT...
Definition: cu-vector.cc:544
CuMatrix< BaseFloat > w_r_m_
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
float BaseFloat
Definition: kaldi-types.h:29
CuVector< BaseFloat > peephole_i_c_
CuSubMatrix< Real > ColRange(const MatrixIndexT col_offset, const MatrixIndexT num_cols) const
Definition: cu-matrix.h:544
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_i_c_corr_
CuVector< BaseFloat > peephole_f_c_
void AddMatMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType transA, const CuMatrixBase< Real > &B, MatrixTransposeType transB, Real beta)
C = alpha * A(^T)*B(^T) + beta * C.
Definition: cu-matrix.cc:1141
int32 proj_dim_
recurrent projection layer dim
CuVector< BaseFloat > peephole_o_c_corr_
CuMatrix< BaseFloat > w_r_m_corr_
CuVector< BaseFloat > bias_corr_
std::vector< int32 > sequence_lengths_
CuMatrix< BaseFloat > w_gifo_x_corr_
CuMatrix< BaseFloat > w_gifo_x_
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 63 of file nnet-lstm-projected.h.

References LstmProjected::LstmProjected().

63 { return new LstmProjected(*this); }
LstmProjected(int32 input_dim, int32 output_dim)
void GetGradient ( VectorBase< BaseFloat > *  gradient) const
inlinevirtual

Get gradient reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 191 of file nnet-lstm-projected.h.

References LstmProjected::bias_, LstmProjected::bias_corr_, VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), LstmProjected::NumParams(), CuMatrixBase< Real >::NumRows(), LstmProjected::peephole_f_c_, LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_, LstmProjected::peephole_o_c_corr_, VectorBase< Real >::Range(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_, LstmProjected::w_gifo_x_corr_, LstmProjected::w_r_m_, and LstmProjected::w_r_m_corr_.

191  {
192  KALDI_ASSERT(gradient->Dim() == NumParams());
193  int32 offset, len;
194 
195  offset = 0; len = w_gifo_x_.NumRows() * w_gifo_x_.NumCols();
196  gradient->Range(offset, len).CopyRowsFromMat(w_gifo_x_corr_);
197 
198  offset += len; len = w_gifo_r_.NumRows() * w_gifo_r_.NumCols();
199  gradient->Range(offset, len).CopyRowsFromMat(w_gifo_r_corr_);
200 
201  offset += len; len = bias_.Dim();
202  gradient->Range(offset, len).CopyFromVec(bias_corr_);
203 
204  offset += len; len = peephole_i_c_.Dim();
205  gradient->Range(offset, len).CopyFromVec(peephole_i_c_corr_);
206 
207  offset += len; len = peephole_f_c_.Dim();
208  gradient->Range(offset, len).CopyFromVec(peephole_f_c_corr_);
209 
210  offset += len; len = peephole_o_c_.Dim();
211  gradient->Range(offset, len).CopyFromVec(peephole_o_c_corr_);
212 
213  offset += len; len = w_r_m_.NumRows() * w_r_m_.NumCols();
214  gradient->Range(offset, len).CopyRowsFromMat(w_r_m_corr_);
215 
216  offset += len;
217  KALDI_ASSERT(offset == NumParams());
218  }
CuVector< BaseFloat > peephole_f_c_corr_
CuMatrix< BaseFloat > w_gifo_r_corr_
CuVector< BaseFloat > peephole_o_c_
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
CuMatrix< BaseFloat > w_r_m_
int32 NumParams() const
Number of trainable parameters,.
CuVector< BaseFloat > peephole_i_c_
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_i_c_corr_
CuVector< BaseFloat > peephole_f_c_
CuVector< BaseFloat > peephole_o_c_corr_
CuMatrix< BaseFloat > w_r_m_corr_
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuVector< BaseFloat > bias_corr_
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
CuMatrix< BaseFloat > w_gifo_x_corr_
CuMatrix< BaseFloat > w_gifo_x_
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void GetParams ( VectorBase< BaseFloat > *  params) const
inlinevirtual

Get the trainable parameters reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 220 of file nnet-lstm-projected.h.

References LstmProjected::bias_, VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), LstmProjected::NumParams(), CuMatrixBase< Real >::NumRows(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, VectorBase< Real >::Range(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

220  {
221  KALDI_ASSERT(params->Dim() == NumParams());
222  int32 offset, len;
223 
224  offset = 0; len = w_gifo_x_.NumRows() * w_gifo_x_.NumCols();
225  params->Range(offset, len).CopyRowsFromMat(w_gifo_x_);
226 
227  offset += len; len = w_gifo_r_.NumRows() * w_gifo_r_.NumCols();
228  params->Range(offset, len).CopyRowsFromMat(w_gifo_r_);
229 
230  offset += len; len = bias_.Dim();
231  params->Range(offset, len).CopyFromVec(bias_);
232 
233  offset += len; len = peephole_i_c_.Dim();
234  params->Range(offset, len).CopyFromVec(peephole_i_c_);
235 
236  offset += len; len = peephole_f_c_.Dim();
237  params->Range(offset, len).CopyFromVec(peephole_f_c_);
238 
239  offset += len; len = peephole_o_c_.Dim();
240  params->Range(offset, len).CopyFromVec(peephole_o_c_);
241 
242  offset += len; len = w_r_m_.NumRows() * w_r_m_.NumCols();
243  params->Range(offset, len).CopyRowsFromMat(w_r_m_);
244 
245  offset += len;
246  KALDI_ASSERT(offset == NumParams());
247  }
CuVector< BaseFloat > peephole_o_c_
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
CuMatrix< BaseFloat > w_r_m_
int32 NumParams() const
Number of trainable parameters,.
CuVector< BaseFloat > peephole_i_c_
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_f_c_
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
CuMatrix< BaseFloat > w_gifo_x_
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 64 of file nnet-lstm-projected.h.

References Component::kLstmProjected.

std::string Info ( ) const
inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 278 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, UpdatableComponent::learn_rate_coef_, kaldi::nnet1::MomentStatistics(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, kaldi::nnet1::ToString(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

278  {
279  return std::string("cell-dim ") + ToString(cell_dim_) + " " +
280  "( learn_rate_coef_ " + ToString(learn_rate_coef_) +
281  ", bias_learn_rate_coef_ " + ToString(bias_learn_rate_coef_) +
282  ", cell_clip_ " + ToString(cell_clip_) +
283  ", diff_clip_ " + ToString(diff_clip_) +
284  ", grad_clip_ " + ToString(grad_clip_) + " )" +
285  "\n w_gifo_x_ " + MomentStatistics(w_gifo_x_) +
286  "\n w_gifo_r_ " + MomentStatistics(w_gifo_r_) +
287  "\n bias_ " + MomentStatistics(bias_) +
288  "\n peephole_i_c_ " + MomentStatistics(peephole_i_c_) +
289  "\n peephole_f_c_ " + MomentStatistics(peephole_f_c_) +
290  "\n peephole_o_c_ " + MomentStatistics(peephole_o_c_) +
291  "\n w_r_m_ " + MomentStatistics(w_r_m_);
292  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
CuVector< BaseFloat > peephole_o_c_
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
CuMatrix< BaseFloat > w_r_m_
CuVector< BaseFloat > peephole_i_c_
BaseFloat grad_clip_
Clipping of the updates,.
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_f_c_
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
CuMatrix< BaseFloat > w_gifo_x_
std::string InfoGradient ( ) const
inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 294 of file nnet-lstm-projected.h.

References LstmProjected::backpropagate_buf_, LstmProjected::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_dim_, CuMatrixBase< Real >::ColRange(), LstmProjected::diff_clip_, LstmProjected::grad_clip_, UpdatableComponent::learn_rate_coef_, kaldi::nnet1::MomentStatistics(), LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_corr_, LstmProjected::proj_dim_, LstmProjected::propagate_buf_, kaldi::nnet1::ToString(), LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_corr_, and LstmProjected::w_r_m_corr_.

294  {
295  // disassemble forward-propagation buffer into different neurons,
296  const CuSubMatrix<BaseFloat> YG(propagate_buf_.ColRange(0*cell_dim_, cell_dim_));
297  const CuSubMatrix<BaseFloat> YI(propagate_buf_.ColRange(1*cell_dim_, cell_dim_));
298  const CuSubMatrix<BaseFloat> YF(propagate_buf_.ColRange(2*cell_dim_, cell_dim_));
299  const CuSubMatrix<BaseFloat> YO(propagate_buf_.ColRange(3*cell_dim_, cell_dim_));
300  const CuSubMatrix<BaseFloat> YC(propagate_buf_.ColRange(4*cell_dim_, cell_dim_));
301  const CuSubMatrix<BaseFloat> YH(propagate_buf_.ColRange(5*cell_dim_, cell_dim_));
302  const CuSubMatrix<BaseFloat> YM(propagate_buf_.ColRange(6*cell_dim_, cell_dim_));
303  const CuSubMatrix<BaseFloat> YR(propagate_buf_.ColRange(7*cell_dim_, proj_dim_));
304 
305  // disassemble backpropagate buffer into different neurons,
306  const CuSubMatrix<BaseFloat> DG(backpropagate_buf_.ColRange(0*cell_dim_, cell_dim_));
307  const CuSubMatrix<BaseFloat> DI(backpropagate_buf_.ColRange(1*cell_dim_, cell_dim_));
308  const CuSubMatrix<BaseFloat> DF(backpropagate_buf_.ColRange(2*cell_dim_, cell_dim_));
309  const CuSubMatrix<BaseFloat> DO(backpropagate_buf_.ColRange(3*cell_dim_, cell_dim_));
310  const CuSubMatrix<BaseFloat> DC(backpropagate_buf_.ColRange(4*cell_dim_, cell_dim_));
311  const CuSubMatrix<BaseFloat> DH(backpropagate_buf_.ColRange(5*cell_dim_, cell_dim_));
312  const CuSubMatrix<BaseFloat> DM(backpropagate_buf_.ColRange(6*cell_dim_, cell_dim_));
313  const CuSubMatrix<BaseFloat> DR(backpropagate_buf_.ColRange(7*cell_dim_, proj_dim_));
314 
315  return std::string("") +
316  "( learn_rate_coef_ " + ToString(learn_rate_coef_) +
317  ", bias_learn_rate_coef_ " + ToString(bias_learn_rate_coef_) +
318  ", cell_clip_ " + ToString(cell_clip_) +
319  ", diff_clip_ " + ToString(diff_clip_) +
320  ", grad_clip_ " + ToString(grad_clip_) + " )" +
321  "\n ### Gradients " +
322  "\n w_gifo_x_corr_ " + MomentStatistics(w_gifo_x_corr_) +
323  "\n w_gifo_r_corr_ " + MomentStatistics(w_gifo_r_corr_) +
324  "\n bias_corr_ " + MomentStatistics(bias_corr_) +
325  "\n peephole_i_c_corr_ " + MomentStatistics(peephole_i_c_corr_) +
326  "\n peephole_f_c_corr_ " + MomentStatistics(peephole_f_c_corr_) +
327  "\n peephole_o_c_corr_ " + MomentStatistics(peephole_o_c_corr_) +
328  "\n w_r_m_corr_ " + MomentStatistics(w_r_m_corr_) +
329  "\n ### Activations (mostly after non-linearities)" +
330  "\n YI(0..1)^ " + MomentStatistics(YI) +
331  "\n YF(0..1)^ " + MomentStatistics(YF) +
332  "\n YO(0..1)^ " + MomentStatistics(YO) +
333  "\n YG(-1..1) " + MomentStatistics(YG) +
334  "\n YC(-R..R)* " + MomentStatistics(YC) +
335  "\n YH(-1..1) " + MomentStatistics(YH) +
336  "\n YM(-1..1) " + MomentStatistics(YM) +
337  "\n YR(-R..R) " + MomentStatistics(YR) +
338  "\n ### Derivatives (w.r.t. inputs of non-linearities)" +
339  "\n DI^ " + MomentStatistics(DI) +
340  "\n DF^ " + MomentStatistics(DF) +
341  "\n DO^ " + MomentStatistics(DO) +
342  "\n DG " + MomentStatistics(DG) +
343  "\n DC* " + MomentStatistics(DC) +
344  "\n DH " + MomentStatistics(DH) +
345  "\n DM " + MomentStatistics(DM) +
346  "\n DR " + MomentStatistics(DR);
347  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
CuMatrix< BaseFloat > backpropagate_buf_
CuVector< BaseFloat > peephole_f_c_corr_
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
CuMatrix< BaseFloat > propagate_buf_
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
CuMatrix< BaseFloat > w_gifo_r_corr_
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
CuSubMatrix< Real > ColRange(const MatrixIndexT col_offset, const MatrixIndexT num_cols) const
Definition: cu-matrix.h:544
BaseFloat grad_clip_
Clipping of the updates,.
CuVector< BaseFloat > peephole_i_c_corr_
int32 proj_dim_
recurrent projection layer dim
CuVector< BaseFloat > peephole_o_c_corr_
CuMatrix< BaseFloat > w_r_m_corr_
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
CuVector< BaseFloat > bias_corr_
CuMatrix< BaseFloat > w_gifo_x_corr_
void InitData ( std::istream &  is)
inlinevirtual

Initialize the content of the component by the 'line' from the prototype,.

Implements UpdatableComponent.

Definition at line 66 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, Component::input_dim_, KALDI_ASSERT, KALDI_ERR, kaldi::kUndefined, UpdatableComponent::learn_rate_coef_, LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::proj_dim_, kaldi::nnet1::RandUniform(), kaldi::ReadBasicType(), kaldi::ReadToken(), CuVector< Real >::Resize(), CuMatrix< Real >::Resize(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

66  {
67  // define options,
68  float param_range = 0.1;
69  // parse the line from prototype,
70  std::string token;
71  while (is >> std::ws, !is.eof()) {
72  ReadToken(is, false, &token);
73  if (token == "<ParamRange>") ReadBasicType(is, false, &param_range);
74  else if (token == "<CellDim>") ReadBasicType(is, false, &cell_dim_);
75  else if (token == "<LearnRateCoef>") ReadBasicType(is, false, &learn_rate_coef_);
76  else if (token == "<BiasLearnRateCoef>") ReadBasicType(is, false, &bias_learn_rate_coef_);
77  else if (token == "<CellClip>") ReadBasicType(is, false, &cell_clip_);
78  else if (token == "<DiffClip>") ReadBasicType(is, false, &diff_clip_);
79  else if (token == "<CellDiffClip>") ReadBasicType(is, false, &cell_diff_clip_);
80  else if (token == "<GradClip>") ReadBasicType(is, false, &grad_clip_);
81  else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
82  << " (ParamRange|CellDim|LearnRateCoef|BiasLearnRateCoef|CellClip|DiffClip|GradClip)";
83  }
84 
85  // init the weights and biases (from uniform dist.),
93  // (mean), (range)
94  RandUniform(0.0, 2.0 * param_range, &w_gifo_x_);
95  RandUniform(0.0, 2.0 * param_range, &w_gifo_r_);
96  RandUniform(0.0, 2.0 * param_range, &bias_);
97  RandUniform(0.0, 2.0 * param_range, &peephole_i_c_);
98  RandUniform(0.0, 2.0 * param_range, &peephole_f_c_);
99  RandUniform(0.0, 2.0 * param_range, &peephole_o_c_);
100  RandUniform(0.0, 2.0 * param_range, &w_r_m_);
101 
102  KALDI_ASSERT(cell_dim_ > 0);
105  }
BaseFloat cell_diff_clip_
Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.
int32 input_dim_
Data members,.
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void RandUniform(BaseFloat mu, BaseFloat range, CuMatrixBase< Real > *mat, struct RandomState *state=NULL)
Fill CuMatrix with random numbers (Uniform distribution): mu = the mean value, range = the 'width' of...
Definition: nnet-utils.h:188
CuVector< BaseFloat > peephole_o_c_
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
CuMatrix< BaseFloat > w_r_m_
CuVector< BaseFloat > peephole_i_c_
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
BaseFloat grad_clip_
Clipping of the updates,.
CuMatrix< BaseFloat > w_gifo_r_
#define KALDI_ERR
Definition: kaldi-error.h:127
CuVector< BaseFloat > peephole_f_c_
int32 proj_dim_
recurrent projection layer dim
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuMatrix< BaseFloat > w_gifo_x_
int32 NumParams ( ) const
inlinevirtual

Number of trainable parameters,.

Implements UpdatableComponent.

Definition at line 181 of file nnet-lstm-projected.h.

References LstmProjected::bias_, CuVectorBase< Real >::Dim(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

Referenced by LstmProjected::GetGradient(), LstmProjected::GetParams(), and LstmProjected::SetParams().

181  {
182  return ( w_gifo_x_.NumRows() * w_gifo_x_.NumCols() +
184  bias_.Dim() +
185  peephole_i_c_.Dim() +
186  peephole_f_c_.Dim() +
187  peephole_o_c_.Dim() +
188  w_r_m_.NumRows() * w_r_m_.NumCols() );
189  }
CuVector< BaseFloat > peephole_o_c_
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
CuMatrix< BaseFloat > w_r_m_
CuVector< BaseFloat > peephole_i_c_
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_f_c_
CuMatrix< BaseFloat > w_gifo_x_
void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 365 of file nnet-lstm-projected.h.

References CuMatrixBase< Real >::AddMatMat(), LstmProjected::bias_, LstmProjected::cell_clip_, LstmProjected::cell_dim_, CuMatrixBase< Real >::ColRange(), CuMatrixBase< Real >::CopyFromMat(), KALDI_ASSERT, kaldi::kNoTrans, kaldi::kSetZero, kaldi::kTrans, CuMatrixBase< Real >::NumRows(), MultistreamComponent::NumStreams(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::prev_nnet_state_, LstmProjected::proj_dim_, LstmProjected::propagate_buf_, LstmProjected::ResetStreams(), CuMatrix< Real >::Resize(), CuMatrixBase< Real >::RowRange(), MultistreamComponent::sequence_lengths_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

366  {
367 
368  // reset context on each sentence if 'sequence_lengths_' not set
369  // (happens in 'nnet-forward' or 'single-stream' training),
370  if (sequence_lengths_.size() == 0) {
371  ResetStreams(std::vector<int32>(1, 1));
372  }
373 
374  KALDI_ASSERT(in.NumRows() % NumStreams() == 0);
375  int32 T = in.NumRows() / NumStreams();
376  int32 S = NumStreams();
377 
378  // buffers,
380  if (prev_nnet_state_.NumRows() != NumStreams()) {
382  } else {
383  propagate_buf_.RowRange(0, S).CopyFromMat(prev_nnet_state_); // use the 'previous-state',
384  }
385 
386  // split activations by neuron types,
387  CuSubMatrix<BaseFloat> YG(propagate_buf_.ColRange(0*cell_dim_, cell_dim_));
388  CuSubMatrix<BaseFloat> YI(propagate_buf_.ColRange(1*cell_dim_, cell_dim_));
389  CuSubMatrix<BaseFloat> YF(propagate_buf_.ColRange(2*cell_dim_, cell_dim_));
390  CuSubMatrix<BaseFloat> YO(propagate_buf_.ColRange(3*cell_dim_, cell_dim_));
391  CuSubMatrix<BaseFloat> YC(propagate_buf_.ColRange(4*cell_dim_, cell_dim_));
392  CuSubMatrix<BaseFloat> YH(propagate_buf_.ColRange(5*cell_dim_, cell_dim_));
393  CuSubMatrix<BaseFloat> YM(propagate_buf_.ColRange(6*cell_dim_, cell_dim_));
394  CuSubMatrix<BaseFloat> YR(propagate_buf_.ColRange(7*cell_dim_, proj_dim_));
395  CuSubMatrix<BaseFloat> YGIFO(propagate_buf_.ColRange(0, 4*cell_dim_));
396 
397  // x -> g, i, f, o, not recurrent, do it all in once
398  YGIFO.RowRange(1*S, T*S).AddMatMat(1.0, in, kNoTrans, w_gifo_x_, kTrans, 0.0);
399 
400  // bias -> g, i, f, o
401  YGIFO.RowRange(1*S, T*S).AddVecToRows(1.0, bias_);
402 
403  // BufferPadding [T0]:dummy, [1, T]:current sequence, [T+1]:dummy
404  for (int t = 1; t <= T; t++) {
405  // multistream buffers for current time-step,
406  CuSubMatrix<BaseFloat> y_all(propagate_buf_.RowRange(t*S, S));
407  CuSubMatrix<BaseFloat> y_g(YG.RowRange(t*S, S));
408  CuSubMatrix<BaseFloat> y_i(YI.RowRange(t*S, S));
409  CuSubMatrix<BaseFloat> y_f(YF.RowRange(t*S, S));
410  CuSubMatrix<BaseFloat> y_o(YO.RowRange(t*S, S));
411  CuSubMatrix<BaseFloat> y_c(YC.RowRange(t*S, S));
412  CuSubMatrix<BaseFloat> y_h(YH.RowRange(t*S, S));
413  CuSubMatrix<BaseFloat> y_m(YM.RowRange(t*S, S));
414  CuSubMatrix<BaseFloat> y_r(YR.RowRange(t*S, S));
415  CuSubMatrix<BaseFloat> y_gifo(YGIFO.RowRange(t*S, S));
416 
417  // r(t-1) -> g, i, f, o
418  y_gifo.AddMatMat(1.0, YR.RowRange((t-1)*S, S), kNoTrans, w_gifo_r_, kTrans, 1.0);
419 
420  // c(t-1) -> i(t) via peephole
421  y_i.AddMatDiagVec(1.0, YC.RowRange((t-1)*S, S), kNoTrans, peephole_i_c_, 1.0);
422 
423  // c(t-1) -> f(t) via peephole
424  y_f.AddMatDiagVec(1.0, YC.RowRange((t-1)*S, S), kNoTrans, peephole_f_c_, 1.0);
425 
426  // i, f sigmoid squashing
427  y_i.Sigmoid(y_i);
428  y_f.Sigmoid(y_f);
429 
430  // g tanh squashing
431  y_g.Tanh(y_g);
432 
433  // g * i -> c
434  y_c.AddMatMatElements(1.0, y_g, y_i, 0.0);
435  // c(t-1) * f -> c(t) via forget-gate
436  y_c.AddMatMatElements(1.0, YC.RowRange((t-1)*S, S), y_f, 1.0);
437 
438  if (cell_clip_ > 0.0) {
439  y_c.ApplyFloor(-cell_clip_); // optional clipping of cell activation,
440  y_c.ApplyCeiling(cell_clip_); // google paper Interspeech2014: LSTM for LVCSR
441  }
442 
443  // c(t) -> o(t) via peephole (non-recurrent, using c(t))
444  y_o.AddMatDiagVec(1.0, y_c, kNoTrans, peephole_o_c_, 1.0);
445 
446  // o sigmoid squashing,
447  y_o.Sigmoid(y_o);
448 
449  // h tanh squashing,
450  y_h.Tanh(y_c);
451 
452  // h * o -> m via output gate,
453  y_m.AddMatMatElements(1.0, y_h, y_o, 0.0);
454 
455  // m -> r
456  y_r.AddMatMat(1.0, y_m, kNoTrans, w_r_m_, kTrans, 0.0);
457 
458  // set zeros to padded frames,
459  if (sequence_lengths_.size() > 0) {
460  for (int s = 0; s < S; s++) {
461  if (t > sequence_lengths_[s]) {
462  y_all.Row(s).SetZero();
463  }
464  }
465  }
466  }
467 
468  // set the 'projection layer' output as the LSTM output,
469  out->CopyFromMat(YR.RowRange(1*S, T*S));
470 
471  // the state in the last 'frame' is transferred (can be zero vector)
473  }
void ResetStreams(const std::vector< int32 > &stream_reset_flag)
TODO: Do we really need this?
CuMatrix< BaseFloat > propagate_buf_
CuVector< BaseFloat > peephole_o_c_
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
CuSubMatrix< Real > RowRange(const MatrixIndexT row_offset, const MatrixIndexT num_rows) const
Definition: cu-matrix.h:539
CuMatrix< BaseFloat > w_r_m_
CuVector< BaseFloat > peephole_i_c_
CuSubMatrix< Real > ColRange(const MatrixIndexT col_offset, const MatrixIndexT num_cols) const
Definition: cu-matrix.h:544
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_f_c_
int32 proj_dim_
recurrent projection layer dim
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuMatrix< BaseFloat > prev_nnet_state_
std::vector< int32 > sequence_lengths_
CuMatrix< BaseFloat > w_gifo_x_
void ReadData ( std::istream &  is,
bool  binary 
)
inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 107 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, kaldi::ExpectToken(), LstmProjected::grad_clip_, KALDI_ASSERT, KALDI_ERR, UpdatableComponent::learn_rate_coef_, kaldi::Peek(), kaldi::PeekToken(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, CuVector< Real >::Read(), CuMatrix< Real >::Read(), kaldi::ReadBasicType(), kaldi::ReadToken(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

107  {
108  // Read all the '<Tokens>' in arbitrary order,
109  while ('<' == Peek(is, binary)) {
110  std::string token;
111  int first_char = PeekToken(is, binary);
112  switch (first_char) {
113  case 'C': ReadToken(is, false, &token);
114  if (token == "<CellDim>") ReadBasicType(is, binary, &cell_dim_);
115  else if (token == "<CellClip>") ReadBasicType(is, binary, &cell_clip_);
116  else if (token == "<CellDiffClip>") ReadBasicType(is, binary, &cell_diff_clip_);
117  else if (token == "<ClipGradient>") ReadBasicType(is, binary, &grad_clip_); // bwd-compat.
118  else KALDI_ERR << "Unknown token: " << token;
119  break;
120  case 'L': ExpectToken(is, binary, "<LearnRateCoef>");
121  ReadBasicType(is, binary, &learn_rate_coef_);
122  break;
123  case 'B': ExpectToken(is, binary, "<BiasLearnRateCoef>");
124  ReadBasicType(is, binary, &bias_learn_rate_coef_);
125  break;
126  case 'D': ExpectToken(is, binary, "<DiffClip>");
127  ReadBasicType(is, binary, &diff_clip_);
128  break;
129  case 'G': ExpectToken(is, binary, "<GradClip>");
130  ReadBasicType(is, binary, &grad_clip_);
131  break;
132  default: ReadToken(is, false, &token);
133  KALDI_ERR << "Unknown token: " << token;
134  }
135  }
136  KALDI_ASSERT(cell_dim_ != 0);
137 
138  // Read the model parameters,
139  w_gifo_x_.Read(is, binary);
140  w_gifo_r_.Read(is, binary);
141  bias_.Read(is, binary);
142 
143  peephole_i_c_.Read(is, binary);
144  peephole_f_c_.Read(is, binary);
145  peephole_o_c_.Read(is, binary);
146 
147  w_r_m_.Read(is, binary);
148  }
BaseFloat cell_diff_clip_
Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Read(std::istream &is, bool binary)
I/O.
Definition: cu-vector.cc:862
CuVector< BaseFloat > peephole_o_c_
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
CuMatrix< BaseFloat > w_r_m_
CuVector< BaseFloat > peephole_i_c_
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
BaseFloat grad_clip_
Clipping of the updates,.
CuMatrix< BaseFloat > w_gifo_r_
#define KALDI_ERR
Definition: kaldi-error.h:127
CuVector< BaseFloat > peephole_f_c_
int PeekToken(std::istream &is, bool binary)
PeekToken will return the first character of the next token, or -1 if end of file.
Definition: io-funcs.cc:170
void Read(std::istream &is, bool binary)
I/O functions.
Definition: cu-matrix.cc:459
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuMatrix< BaseFloat > w_gifo_x_
void ResetStreams ( const std::vector< int32 > &  stream_reset_flag)
inlinevirtual

TODO: Do we really need this?

Reimplemented from MultistreamComponent.

Definition at line 352 of file nnet-lstm-projected.h.

References LstmProjected::cell_dim_, KALDI_ASSERT, kaldi::kSetZero, CuMatrixBase< Real >::NumRows(), MultistreamComponent::NumStreams(), LstmProjected::prev_nnet_state_, LstmProjected::proj_dim_, CuMatrix< Real >::Resize(), and CuMatrixBase< Real >::Row().

Referenced by LstmProjected::PropagateFnc().

352  {
353  KALDI_ASSERT(NumStreams() == stream_reset_flag.size());
354  if (prev_nnet_state_.NumRows() != stream_reset_flag.size()) {
356  } else {
357  for (int s = 0; s < NumStreams(); s++) {
358  if (stream_reset_flag[s] == 1) {
359  prev_nnet_state_.Row(s).SetZero();
360  }
361  }
362  }
363  }
const CuSubVector< Real > Row(MatrixIndexT i) const
Definition: cu-matrix.h:549
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
int32 proj_dim_
recurrent projection layer dim
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuMatrix< BaseFloat > prev_nnet_state_
void SetParams ( const VectorBase< BaseFloat > &  params)
inlinevirtual

Set the trainable parameters from, reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 249 of file nnet-lstm-projected.h.

References LstmProjected::bias_, CuVectorBase< Real >::CopyFromVec(), CuMatrixBase< Real >::CopyRowsFromVec(), VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), LstmProjected::NumParams(), CuMatrixBase< Real >::NumRows(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, VectorBase< Real >::Range(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

249  {
250  KALDI_ASSERT(params.Dim() == NumParams());
251  int32 offset, len;
252 
253  offset = 0; len = w_gifo_x_.NumRows() * w_gifo_x_.NumCols();
254  w_gifo_x_.CopyRowsFromVec(params.Range(offset, len));
255 
256  offset += len; len = w_gifo_r_.NumRows() * w_gifo_r_.NumCols();
257  w_gifo_r_.CopyRowsFromVec(params.Range(offset, len));
258 
259  offset += len; len = bias_.Dim();
260  bias_.CopyFromVec(params.Range(offset, len));
261 
262  offset += len; len = peephole_i_c_.Dim();
263  peephole_i_c_.CopyFromVec(params.Range(offset, len));
264 
265  offset += len; len = peephole_f_c_.Dim();
266  peephole_f_c_.CopyFromVec(params.Range(offset, len));
267 
268  offset += len; len = peephole_o_c_.Dim();
269  peephole_o_c_.CopyFromVec(params.Range(offset, len));
270 
271  offset += len; len = w_r_m_.NumRows() * w_r_m_.NumCols();
272  w_r_m_.CopyRowsFromVec(params.Range(offset, len));
273 
274  offset += len;
275  KALDI_ASSERT(offset == NumParams());
276  }
void CopyRowsFromVec(const CuVectorBase< Real > &v)
This function has two modes of operation.
Definition: cu-matrix.cc:2144
CuVector< BaseFloat > peephole_o_c_
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
CuMatrix< BaseFloat > w_r_m_
int32 NumParams() const
Number of trainable parameters,.
CuVector< BaseFloat > peephole_i_c_
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > w_gifo_r_
void CopyFromVec(const CuVectorBase< Real > &src)
Copy functions; these will crash if the dimension do not match.
Definition: cu-vector.cc:970
CuVector< BaseFloat > peephole_f_c_
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
CuMatrix< BaseFloat > w_gifo_x_
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void Update ( const CuMatrixBase< BaseFloat > &  input,
const CuMatrixBase< BaseFloat > &  diff 
)
inlinevirtual

Compute gradient and update parameters,.

Implements UpdatableComponent.

Definition at line 654 of file nnet-lstm-projected.h.

References CuMatrixBase< Real >::AddMat(), CuVectorBase< Real >::AddVec(), CuVectorBase< Real >::ApplyCeiling(), CuMatrixBase< Real >::ApplyCeiling(), CuVectorBase< Real >::ApplyFloor(), CuMatrixBase< Real >::ApplyFloor(), LstmProjected::bias_, LstmProjected::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::clip_gradient_, NnetTrainOptions::learn_rate, UpdatableComponent::learn_rate_coef_, UpdatableComponent::opts_, LstmProjected::peephole_f_c_, LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_, LstmProjected::peephole_o_c_corr_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_, LstmProjected::w_gifo_x_corr_, LstmProjected::w_r_m_, and LstmProjected::w_r_m_corr_.

655  {
656 
657  // apply the gradient clipping,
658  if (clip_gradient_ > 0.0) {
673  }
674 
675  const BaseFloat lr = opts_.learn_rate;
676 
678  w_gifo_r_.AddMat(-lr * learn_rate_coef_, w_gifo_r_corr_);
680 
681  peephole_i_c_.AddVec(-lr * bias_learn_rate_coef_, peephole_i_c_corr_, 1.0);
682  peephole_f_c_.AddVec(-lr * bias_learn_rate_coef_, peephole_f_c_corr_, 1.0);
683  peephole_o_c_.AddVec(-lr * bias_learn_rate_coef_, peephole_o_c_corr_, 1.0);
684 
685  w_r_m_.AddMat(-lr * learn_rate_coef_, w_r_m_corr_);
686  }
CuVector< BaseFloat > peephole_f_c_corr_
NnetTrainOptions opts_
Option-class with training hyper-parameters,.
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
CuMatrix< BaseFloat > w_gifo_r_corr_
void ApplyCeiling(Real ceiling_val)
Definition: cu-matrix.cc:2383
CuVector< BaseFloat > peephole_o_c_
CuMatrix< BaseFloat > w_r_m_
void ApplyFloor(Real floor_val)
Definition: cu-matrix.cc:2365
float BaseFloat
Definition: kaldi-types.h:29
CuVector< BaseFloat > peephole_i_c_
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_i_c_corr_
CuVector< BaseFloat > peephole_f_c_
CuVector< BaseFloat > peephole_o_c_corr_
CuMatrix< BaseFloat > w_r_m_corr_
void AddMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType trans=kNoTrans)
*this += alpha * A
Definition: cu-matrix.cc:939
MatrixIndexT ApplyFloor(Real floor_val)
Definition: cu-vector.cc:324
CuVector< BaseFloat > bias_corr_
MatrixIndexT ApplyCeiling(Real ceiling_val)
Definition: cu-vector.cc:349
void AddVec(Real alpha, const CuVectorBase< Real > &vec, Real beta=1.0)
Definition: cu-vector.cc:1126
CuMatrix< BaseFloat > w_gifo_x_corr_
CuMatrix< BaseFloat > w_gifo_x_
void WriteData ( std::ostream &  os,
bool  binary 
) const
inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 150 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, UpdatableComponent::learn_rate_coef_, LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, LstmProjected::w_r_m_, CuVector< Real >::Write(), CuMatrixBase< Real >::Write(), kaldi::WriteBasicType(), and kaldi::WriteToken().

150  {
151  WriteToken(os, binary, "<CellDim>");
152  WriteBasicType(os, binary, cell_dim_);
153 
154  WriteToken(os, binary, "<LearnRateCoef>");
155  WriteBasicType(os, binary, learn_rate_coef_);
156  WriteToken(os, binary, "<BiasLearnRateCoef>");
158 
159  WriteToken(os, binary, "<CellClip>");
160  WriteBasicType(os, binary, cell_clip_);
161  WriteToken(os, binary, "<DiffClip>");
162  WriteBasicType(os, binary, diff_clip_);
163  WriteToken(os, binary, "<CellDiffClip>");
164  WriteBasicType(os, binary, cell_diff_clip_);
165  WriteToken(os, binary, "<GradClip>");
166  WriteBasicType(os, binary, grad_clip_);
167 
168  // write model parameters,
169  if (!binary) os << "\n";
170  w_gifo_x_.Write(os, binary);
171  w_gifo_r_.Write(os, binary);
172  bias_.Write(os, binary);
173 
174  peephole_i_c_.Write(os, binary);
175  peephole_f_c_.Write(os, binary);
176  peephole_o_c_.Write(os, binary);
177 
178  w_r_m_.Write(os, binary);
179  }
BaseFloat cell_diff_clip_
Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
CuVector< BaseFloat > peephole_o_c_
BaseFloat diff_clip_
Clipping of 'derivatives' in backprop (per-frame),.
CuMatrix< BaseFloat > w_r_m_
void Write(std::ostream &is, bool binary) const
Definition: cu-vector.cc:872
CuVector< BaseFloat > peephole_i_c_
BaseFloat grad_clip_
Clipping of the updates,.
CuMatrix< BaseFloat > w_gifo_r_
CuVector< BaseFloat > peephole_f_c_
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
BaseFloat cell_clip_
Clipping of 'cell-values' in forward pass (per-frame),.
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
CuMatrix< BaseFloat > w_gifo_x_
void Write(std::ostream &os, bool binary) const
Definition: cu-matrix.cc:467

Member Data Documentation

CuMatrix<BaseFloat> backpropagate_buf_
private
BaseFloat cell_clip_
private
BaseFloat cell_diff_clip_
private

Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.

Definition at line 695 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::InitData(), LstmProjected::ReadData(), and LstmProjected::WriteData().

BaseFloat clip_gradient_
private

Definition at line 702 of file nnet-lstm-projected.h.

Referenced by LstmProjected::Update().

BaseFloat diff_clip_
private
CuMatrix<BaseFloat> prev_nnet_state_
private
int32 proj_dim_
private

The documentation for this class was generated from the following file: