LstmProjected Class Reference

#include <nnet-lstm-projected.h>

Inheritance diagram for LstmProjected:

Collaboration diagram for LstmProjected:

Public Member Functions
	LstmProjected (int32 input_dim, int32 output_dim)
	~LstmProjected ()
Component *	Copy () const
	Copy component (deep copy),. More...
ComponentType	GetType () const
	Get Type Identification of the component,. More...
void	InitData (std::istream &is)
	Initialize the content of the component by the 'line' from the prototype,. More...
void	ReadData (std::istream &is, bool binary)
	Reads the component content. More...
void	WriteData (std::ostream &os, bool binary) const
	Writes the component content. More...
int32	NumParams () const
	Number of trainable parameters,. More...
void	GetGradient (VectorBase< BaseFloat > *gradient) const
	Get gradient reshaped as a vector,. More...
void	GetParams (VectorBase< BaseFloat > *params) const
	Get the trainable parameters reshaped as a vector,. More...
void	SetParams (const VectorBase< BaseFloat > &params)
	Set the trainable parameters from, reshaped as a vector,. More...
std::string	Info () const
	Print some additional info (after <ComponentName> and the dims),. More...
std::string	InfoGradient () const
	Print some additional info about gradient (after <...> and dims),. More...
void	ResetStreams (const std::vector< int32 > &stream_reset_flag)
	TODO: Do we really need this? More...
void	PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
	Abstract interface for propagation/backpropagation. More...
void	BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
	Backward pass transformation (to be implemented by descending class...) More...
void	Update (const CuMatrixBase< BaseFloat > &input, const CuMatrixBase< BaseFloat > &diff)
	Compute gradient and update parameters,. More...
Public Member Functions inherited from MultistreamComponent
	MultistreamComponent (int32 input_dim, int32 output_dim)
bool	IsMultistream () const
	Check if component has 'Recurrent' interface (trainable and recurrent),. More...
virtual void	SetSeqLengths (const std::vector< int32 > &sequence_lengths)
int32	NumStreams () const
Public Member Functions inherited from UpdatableComponent
	UpdatableComponent (int32 input_dim, int32 output_dim)
virtual	~UpdatableComponent ()
bool	IsUpdatable () const
	Check if contains trainable parameters,. More...
virtual void	SetTrainOptions (const NnetTrainOptions &opts)
	Set the training options to the component,. More...
const NnetTrainOptions &	GetTrainOptions () const
	Get the training options from the component,. More...
virtual void	SetLearnRateCoef (BaseFloat val)
	Set the learn-rate coefficient,. More...
virtual void	SetBiasLearnRateCoef (BaseFloat val)
	Set the learn-rate coefficient for bias,. More...
Public Member Functions inherited from Component
	Component (int32 input_dim, int32 output_dim)
	Generic interface of a component,. More...
virtual	~Component ()
int32	InputDim () const
	Get the dimension of the input,. More...
int32	OutputDim () const
	Get the dimension of the output,. More...
void	Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward-pass propagation 'in' -> 'out',. More...
void	Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
	Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
void	Write (std::ostream &os, bool binary) const
	Write the component to a stream,. More...

Private Attributes
int32	cell_dim_
int32	proj_dim_
	recurrent projection layer dim More...
BaseFloat	cell_clip_
	Clipping of 'cell-values' in forward pass (per-frame),. More...
BaseFloat	diff_clip_
	Clipping of 'derivatives' in backprop (per-frame),. More...
BaseFloat	cell_diff_clip_
	Clipping of 'cell-derivatives' accumulated over CEC (per-frame),. More...
BaseFloat	grad_clip_
	Clipping of the updates,. More...
CuMatrix< BaseFloat >	prev_nnet_state_
CuMatrix< BaseFloat >	w_gifo_x_
CuMatrix< BaseFloat >	w_gifo_x_corr_
CuMatrix< BaseFloat >	w_gifo_r_
CuMatrix< BaseFloat >	w_gifo_r_corr_
CuVector< BaseFloat >	bias_
CuVector< BaseFloat >	bias_corr_
CuVector< BaseFloat >	peephole_i_c_
CuVector< BaseFloat >	peephole_f_c_
CuVector< BaseFloat >	peephole_o_c_
CuVector< BaseFloat >	peephole_i_c_corr_
CuVector< BaseFloat >	peephole_f_c_corr_
CuVector< BaseFloat >	peephole_o_c_corr_
CuMatrix< BaseFloat >	w_r_m_
CuMatrix< BaseFloat >	w_r_m_corr_
CuMatrix< BaseFloat >	propagate_buf_
CuMatrix< BaseFloat >	backpropagate_buf_

Additional Inherited Members
Public Types inherited from Component
enum	ComponentType {   kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,   kConvolutionalComponent, kLstmProjected, kBlstmProjected, kRecurrentComponent,   kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax, kBlockSoftmax,   kSigmoid, kTanh, kParametricRelu, kDropout,   kLengthNormComponent, kTranform = 0x0400, kRbm, kSplice,   kCopy, kTranspose, kBlockLinearity, kAddShift,   kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent, kSimpleSentenceAveragingComponent,   kAveragePoolingComponent, kMaxPoolingComponent, kFramePoolingComponent, kParallelComponent,   kMultiBasisComponent }
	Component type identification mechanism,. More...
Static Public Member Functions inherited from Component
static const char *	TypeToMarker (ComponentType t)
	Converts component type to marker,. More...
static ComponentType	MarkerToType (const std::string &s)
	Converts marker to component type (case insensitive),. More...
static Component *	Init (const std::string &conf_line)
	Initialize component from a line in config file,. More...
static Component *	Read (std::istream &is, bool binary)
	Read the component from a stream (static method),. More...
Static Public Attributes inherited from Component
static const struct key_value	kMarkerMap []
	The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
Protected Attributes inherited from MultistreamComponent
std::vector< int32 >	sequence_lengths_
Protected Attributes inherited from UpdatableComponent
NnetTrainOptions	opts_
	Option-class with training hyper-parameters,. More...
BaseFloat	learn_rate_coef_
	Scalar applied to learning rate for weight matrices (to be used in ::Update method),. More...
BaseFloat	bias_learn_rate_coef_
	Scalar applied to learning rate for bias (to be used in ::Update method),. More...
Protected Attributes inherited from Component
int32	input_dim_
	Data members,. More...
int32	output_dim_
	Dimension of the output of the Component,. More...

Detailed Description

Definition at line 48 of file nnet-lstm-projected.h.

Constructor & Destructor Documentation

LstmProjected()

LstmProjected ( int32  input_dim, int32  output_dim  )

inline

Definition at line 50 of file nnet-lstm-projected.h.

Referenced by LstmProjected::Copy().

                                                  :
    MultistreamComponent(input_dim, output_dim),
    cell_dim_(0),
    proj_dim_(output_dim),
    cell_clip_(50.0),
    diff_clip_(1.0),
    cell_diff_clip_(0.0),
    grad_clip_(250.0)
  { }

~LstmProjected()

~LstmProjected ( )

inline

Definition at line 60 of file nnet-lstm-projected.h.

  { }

Member Function Documentation

BackpropagateFnc()

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in, const CuMatrixBase< BaseFloat > &  out, const CuMatrixBase< BaseFloat > &  out_diff, CuMatrixBase< BaseFloat > *  in_diff  )

inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 475 of file nnet-lstm-projected.h.

References CuMatrixBase< Real >::AddMatMat(), LstmProjected::backpropagate_buf_, LstmProjected::bias_corr_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, Component::input_dim_, kaldi::kNoTrans, kaldi::kSetZero, kaldi::kTrans, NnetTrainOptions::momentum, CuMatrixBase< Real >::NumRows(), MultistreamComponent::NumStreams(), UpdatableComponent::opts_, LstmProjected::peephole_f_c_, LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_, LstmProjected::peephole_o_c_corr_, LstmProjected::proj_dim_, LstmProjected::propagate_buf_, CuMatrixBase< Real >::RowRange(), MultistreamComponent::sequence_lengths_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_, LstmProjected::w_gifo_x_corr_, LstmProjected::w_r_m_, and LstmProjected::w_r_m_corr_.

                                                          {

    // the number of sequences to be processed in parallel
    int32 T = in.NumRows() / NumStreams();
    int32 S = NumStreams();

    // buffer,
    backpropagate_buf_.Resize((T+2)*S, 7 * cell_dim_ + proj_dim_, kSetZero);

    // split activations by neuron types,
    CuSubMatrix<BaseFloat> YG(propagate_buf_.ColRange(0*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YI(propagate_buf_.ColRange(1*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YF(propagate_buf_.ColRange(2*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YO(propagate_buf_.ColRange(3*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YC(propagate_buf_.ColRange(4*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YH(propagate_buf_.ColRange(5*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YM(propagate_buf_.ColRange(6*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YR(propagate_buf_.ColRange(7*cell_dim_, proj_dim_));

    // split derivatives by neuron types,
    CuSubMatrix<BaseFloat> DG(backpropagate_buf_.ColRange(0*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DI(backpropagate_buf_.ColRange(1*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DF(backpropagate_buf_.ColRange(2*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DO(backpropagate_buf_.ColRange(3*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DC(backpropagate_buf_.ColRange(4*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DH(backpropagate_buf_.ColRange(5*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DM(backpropagate_buf_.ColRange(6*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> DR(backpropagate_buf_.ColRange(7*cell_dim_, proj_dim_));
    CuSubMatrix<BaseFloat> DGIFO(backpropagate_buf_.ColRange(0, 4*cell_dim_));

    // pre-copy partial derivatives from the LSTM output,
    DR.RowRange(1*S, T*S).CopyFromMat(out_diff);

    // BufferPadding [T0]:dummy, [1,T]:current sequence, [T+1]: dummy,
    for (int t = T; t >= 1; t--) {
      CuSubMatrix<BaseFloat> y_g(YG.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_i(YI.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_f(YF.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_o(YO.RowRange(t*S, S));
      // CuSubMatrix<BaseFloat> y_c(YC.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_h(YH.RowRange(t*S, S));
      // CuSubMatrix<BaseFloat> y_m(YM.RowRange(t*S, S));
      // CuSubMatrix<BaseFloat> y_r(YR.RowRange(t*S, S));

      CuSubMatrix<BaseFloat> d_all(backpropagate_buf_.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_g(DG.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_i(DI.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_f(DF.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_o(DO.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_c(DC.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_h(DH.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_m(DM.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_r(DR.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> d_gifo(DGIFO.RowRange(t*S, S));

      // r
      //   Version 1 (precise gradients):
      //   backprop error from g(t+1), i(t+1), f(t+1), o(t+1) to r(t)
      d_r.AddMatMat(1.0, DGIFO.RowRange((t+1)*S, S), kNoTrans, w_gifo_r_, kNoTrans, 1.0);

      /*
      //   Version 2 (Alex Graves' PhD dissertation):
      //   only backprop g(t+1) to r(t)
      CuSubMatrix<BaseFloat> w_g_r_(w_gifo_r_.RowRange(0, cell_dim_));
      d_r.AddMatMat(1.0, DG.RowRange((t+1)*S,S), kNoTrans, w_g_r_, kNoTrans, 1.0);
      */

      /*
      //   Version 3 (Felix Gers' PhD dissertation):
      //   truncate gradients of g(t+1), i(t+1), f(t+1), o(t+1) once they leak out memory block
      //   CEC(with forget connection) is the only "error-bridge" through time
      */

      // r -> m
      d_m.AddMatMat(1.0, d_r, kNoTrans, w_r_m_, kNoTrans, 0.0);

      // m -> h via output gate
      d_h.AddMatMatElements(1.0, d_m, y_o, 0.0);
      d_h.DiffTanh(y_h, d_h);

      // o
      d_o.AddMatMatElements(1.0, d_m, y_h, 0.0);
      d_o.DiffSigmoid(y_o, d_o);

      // c
      // 1. diff from h(t)
      // 2. diff from c(t+1) (via forget-gate between CEC)
      // 3. diff from i(t+1) (via peephole)
      // 4. diff from f(t+1) (via peephole)
      // 5. diff from o(t)   (via peephole, not recurrent)
      d_c.AddMat(1.0, d_h);
      d_c.AddMatMatElements(1.0, DC.RowRange((t+1)*S, S), YF.RowRange((t+1)*S,S), 1.0);
      d_c.AddMatDiagVec(1.0, DI.RowRange((t+1)*S, S), kNoTrans, peephole_i_c_, 1.0);
      d_c.AddMatDiagVec(1.0, DF.RowRange((t+1)*S, S), kNoTrans, peephole_f_c_, 1.0);
      d_c.AddMatDiagVec(1.0, d_o                    , kNoTrans, peephole_o_c_, 1.0);
      // optionally clip the cell_derivative,
      if (cell_diff_clip_ > 0.0) {
        d_c.ApplyFloor(-cell_diff_clip_);
        d_c.ApplyCeiling(cell_diff_clip_);
      }

      // f
      d_f.AddMatMatElements(1.0, d_c, YC.RowRange((t-1)*S,S), 0.0);
      d_f.DiffSigmoid(y_f, d_f);

      // i
      d_i.AddMatMatElements(1.0, d_c, y_g, 0.0);
      d_i.DiffSigmoid(y_i, d_i);

      // c -> g via input gate
      d_g.AddMatMatElements(1.0, d_c, y_i, 0.0);
      d_g.DiffTanh(y_g, d_g);

      // Clipping per-frame derivatives for the next `t'.
      // Clipping applied to gates and input gate (as done in Google).
      // [ICASSP2015, Sak, Learning acoustic frame labelling...],
      //
      // The path from 'out_diff' to 'd_c' via 'd_h' is unclipped,
      // which is probably important for the 'Constant Error Carousel'
      // to work well.
      //
      if (diff_clip_ > 0.0) {
        d_gifo.ApplyFloor(-diff_clip_);
        d_gifo.ApplyCeiling(diff_clip_);
      }

      // set zeros to padded frames,
      if (sequence_lengths_.size() > 0) {
        for (int s = 0; s < S; s++) {
          if (t > sequence_lengths_[s]) {
            d_all.Row(s).SetZero();
          }
        }
      }
    }

    // g,i,f,o -> x, calculating input derivatives,
    in_diff->AddMatMat(1.0, DGIFO.RowRange(1*S,T*S), kNoTrans, w_gifo_x_, kNoTrans, 0.0);

    // lazy initialization of udpate buffers,
    if (w_gifo_x_corr_.NumRows() == 0) {
      w_gifo_x_corr_.Resize(4*cell_dim_, input_dim_, kSetZero);
      w_gifo_r_corr_.Resize(4*cell_dim_, proj_dim_, kSetZero);
      bias_corr_.Resize(4*cell_dim_, kSetZero);
      peephole_i_c_corr_.Resize(cell_dim_, kSetZero);
      peephole_f_c_corr_.Resize(cell_dim_, kSetZero);
      peephole_o_c_corr_.Resize(cell_dim_, kSetZero);
      w_r_m_corr_.Resize(proj_dim_, cell_dim_, kSetZero);
    }

    // calculate delta
    const BaseFloat mmt = opts_.momentum;

    // weight x -> g, i, f, o
    w_gifo_x_corr_.AddMatMat(1.0, DGIFO.RowRange(1*S, T*S), kTrans,
                                  in                      , kNoTrans, mmt);
    // recurrent weight r -> g, i, f, o
    w_gifo_r_corr_.AddMatMat(1.0, DGIFO.RowRange(1*S, T*S), kTrans,
                                  YR.RowRange(0*S, T*S)   , kNoTrans, mmt);
    // bias of g, i, f, o
    bias_corr_.AddRowSumMat(1.0, DGIFO.RowRange(1*S, T*S), mmt);

    // recurrent peephole c -> i
    peephole_i_c_corr_.AddDiagMatMat(1.0, DI.RowRange(1*S, T*S), kTrans,
                                          YC.RowRange(0*S, T*S), kNoTrans, mmt);
    // recurrent peephole c -> f
    peephole_f_c_corr_.AddDiagMatMat(1.0, DF.RowRange(1*S, T*S), kTrans,
                                          YC.RowRange(0*S, T*S), kNoTrans, mmt);
    // peephole c -> o
    peephole_o_c_corr_.AddDiagMatMat(1.0, DO.RowRange(1*S, T*S), kTrans,
                                          YC.RowRange(1*S, T*S), kNoTrans, mmt);

    w_r_m_corr_.AddMatMat(1.0, DR.RowRange(1*S, T*S), kTrans,
                               YM.RowRange(1*S, T*S), kNoTrans, mmt);
  }

Copy()

Component* Copy ( ) const

inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 63 of file nnet-lstm-projected.h.

References LstmProjected::LstmProjected().

{ return new LstmProjected(*this); }

GetGradient()

void GetGradient ( VectorBase< BaseFloat > *  gradient ) const

inlinevirtual

Get gradient reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 191 of file nnet-lstm-projected.h.

References LstmProjected::bias_, LstmProjected::bias_corr_, VectorBase< Real >::Dim(), KALDI_ASSERT, LstmProjected::NumParams(), LstmProjected::peephole_f_c_, LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_, LstmProjected::peephole_o_c_corr_, VectorBase< Real >::Range(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_, LstmProjected::w_gifo_x_corr_, LstmProjected::w_r_m_, and LstmProjected::w_r_m_corr_.

                                                          {
    KALDI_ASSERT(gradient->Dim() == NumParams());
    int32 offset, len;

    offset = 0;    len = w_gifo_x_.NumRows() * w_gifo_x_.NumCols();
    gradient->Range(offset, len).CopyRowsFromMat(w_gifo_x_corr_);

    offset += len; len = w_gifo_r_.NumRows() * w_gifo_r_.NumCols();
    gradient->Range(offset, len).CopyRowsFromMat(w_gifo_r_corr_);

    offset += len; len = bias_.Dim();
    gradient->Range(offset, len).CopyFromVec(bias_corr_);

    offset += len; len = peephole_i_c_.Dim();
    gradient->Range(offset, len).CopyFromVec(peephole_i_c_corr_);

    offset += len; len = peephole_f_c_.Dim();
    gradient->Range(offset, len).CopyFromVec(peephole_f_c_corr_);

    offset += len; len = peephole_o_c_.Dim();
    gradient->Range(offset, len).CopyFromVec(peephole_o_c_corr_);

    offset += len; len = w_r_m_.NumRows() * w_r_m_.NumCols();
    gradient->Range(offset, len).CopyRowsFromMat(w_r_m_corr_);

    offset += len;
    KALDI_ASSERT(offset == NumParams());
  }

GetParams()

void GetParams ( VectorBase< BaseFloat > *  params ) const

inlinevirtual

Get the trainable parameters reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 220 of file nnet-lstm-projected.h.

References LstmProjected::bias_, VectorBase< Real >::Dim(), KALDI_ASSERT, LstmProjected::NumParams(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, VectorBase< Real >::Range(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

                                                      {
    KALDI_ASSERT(params->Dim() == NumParams());
    int32 offset, len;

    offset = 0;    len = w_gifo_x_.NumRows() * w_gifo_x_.NumCols();
    params->Range(offset, len).CopyRowsFromMat(w_gifo_x_);

    offset += len; len = w_gifo_r_.NumRows() * w_gifo_r_.NumCols();
    params->Range(offset, len).CopyRowsFromMat(w_gifo_r_);

    offset += len; len = bias_.Dim();
    params->Range(offset, len).CopyFromVec(bias_);

    offset += len; len = peephole_i_c_.Dim();
    params->Range(offset, len).CopyFromVec(peephole_i_c_);

    offset += len; len = peephole_f_c_.Dim();
    params->Range(offset, len).CopyFromVec(peephole_f_c_);

    offset += len; len = peephole_o_c_.Dim();
    params->Range(offset, len).CopyFromVec(peephole_o_c_);

    offset += len; len = w_r_m_.NumRows() * w_r_m_.NumCols();
    params->Range(offset, len).CopyRowsFromMat(w_r_m_);

    offset += len;
    KALDI_ASSERT(offset == NumParams());
  }

GetType()

ComponentType GetType ( ) const

inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 64 of file nnet-lstm-projected.h.

References Component::kLstmProjected.

{ return kLstmProjected; }

Info()

std::string Info ( ) const

inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 278 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, UpdatableComponent::learn_rate_coef_, kaldi::nnet1::MomentStatistics(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, kaldi::nnet1::ToString(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

                         {
    return std::string("cell-dim ") + ToString(cell_dim_) + " " +
      "( learn_rate_coef_ " + ToString(learn_rate_coef_) +
      ", bias_learn_rate_coef_ " + ToString(bias_learn_rate_coef_) +
      ", cell_clip_ " + ToString(cell_clip_) +
      ", diff_clip_ " + ToString(diff_clip_) +
      ", grad_clip_ " + ToString(grad_clip_) + " )" +
      "\n  w_gifo_x_  "   + MomentStatistics(w_gifo_x_) +
      "\n  w_gifo_r_  "   + MomentStatistics(w_gifo_r_) +
      "\n  bias_  "     + MomentStatistics(bias_) +
      "\n  peephole_i_c_  " + MomentStatistics(peephole_i_c_) +
      "\n  peephole_f_c_  " + MomentStatistics(peephole_f_c_) +
      "\n  peephole_o_c_  " + MomentStatistics(peephole_o_c_) +
      "\n  w_r_m_  "    + MomentStatistics(w_r_m_);
  }

InfoGradient()

std::string InfoGradient ( ) const

inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 294 of file nnet-lstm-projected.h.

References LstmProjected::backpropagate_buf_, LstmProjected::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, UpdatableComponent::learn_rate_coef_, kaldi::nnet1::MomentStatistics(), LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_corr_, LstmProjected::proj_dim_, LstmProjected::propagate_buf_, kaldi::nnet1::ToString(), LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_corr_, and LstmProjected::w_r_m_corr_.

                                 {
    // disassemble forward-propagation buffer into different neurons,
    const CuSubMatrix<BaseFloat> YG(propagate_buf_.ColRange(0*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YI(propagate_buf_.ColRange(1*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YF(propagate_buf_.ColRange(2*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YO(propagate_buf_.ColRange(3*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YC(propagate_buf_.ColRange(4*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YH(propagate_buf_.ColRange(5*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YM(propagate_buf_.ColRange(6*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> YR(propagate_buf_.ColRange(7*cell_dim_, proj_dim_));

    // disassemble backpropagate buffer into different neurons,
    const CuSubMatrix<BaseFloat> DG(backpropagate_buf_.ColRange(0*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DI(backpropagate_buf_.ColRange(1*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DF(backpropagate_buf_.ColRange(2*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DO(backpropagate_buf_.ColRange(3*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DC(backpropagate_buf_.ColRange(4*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DH(backpropagate_buf_.ColRange(5*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DM(backpropagate_buf_.ColRange(6*cell_dim_, cell_dim_));
    const CuSubMatrix<BaseFloat> DR(backpropagate_buf_.ColRange(7*cell_dim_, proj_dim_));

    return std::string("") +
      "( learn_rate_coef_ " + ToString(learn_rate_coef_) +
      ", bias_learn_rate_coef_ " + ToString(bias_learn_rate_coef_) +
      ", cell_clip_ " + ToString(cell_clip_) +
      ", diff_clip_ " + ToString(diff_clip_) +
      ", grad_clip_ " + ToString(grad_clip_) + " )" +
      "\n  ### Gradients " +
      "\n  w_gifo_x_corr_  "   + MomentStatistics(w_gifo_x_corr_) +
      "\n  w_gifo_r_corr_  "   + MomentStatistics(w_gifo_r_corr_) +
      "\n  bias_corr_  "     + MomentStatistics(bias_corr_) +
      "\n  peephole_i_c_corr_  " + MomentStatistics(peephole_i_c_corr_) +
      "\n  peephole_f_c_corr_  " + MomentStatistics(peephole_f_c_corr_) +
      "\n  peephole_o_c_corr_  " + MomentStatistics(peephole_o_c_corr_) +
      "\n  w_r_m_corr_  "    + MomentStatistics(w_r_m_corr_) +
      "\n  ### Activations (mostly after non-linearities)" +
      "\n  YI(0..1)^  " + MomentStatistics(YI) +
      "\n  YF(0..1)^  " + MomentStatistics(YF) +
      "\n  YO(0..1)^  " + MomentStatistics(YO) +
      "\n  YG(-1..1)  " + MomentStatistics(YG) +
      "\n  YC(-R..R)* " + MomentStatistics(YC) +
      "\n  YH(-1..1)  " + MomentStatistics(YH) +
      "\n  YM(-1..1)  " + MomentStatistics(YM) +
      "\n  YR(-R..R)  " + MomentStatistics(YR) +
      "\n  ### Derivatives (w.r.t. inputs of non-linearities)" +
      "\n  DI^ " + MomentStatistics(DI) +
      "\n  DF^ " + MomentStatistics(DF) +
      "\n  DO^ " + MomentStatistics(DO) +
      "\n  DG  " + MomentStatistics(DG) +
      "\n  DC* " + MomentStatistics(DC) +
      "\n  DH  " + MomentStatistics(DH) +
      "\n  DM  " + MomentStatistics(DM) +
      "\n  DR  " + MomentStatistics(DR);
  }

InitData()

void InitData ( std::istream &  is )

inlinevirtual

Initialize the content of the component by the 'line' from the prototype,.

Implements UpdatableComponent.

Definition at line 66 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, Component::input_dim_, KALDI_ASSERT, KALDI_ERR, kaldi::kUndefined, UpdatableComponent::learn_rate_coef_, LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::proj_dim_, kaldi::nnet1::RandUniform(), kaldi::ReadBasicType(), kaldi::ReadToken(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

                                {
    // define options,
    float param_range = 0.1;
    // parse the line from prototype,
    std::string token;
    while (is >> std::ws, !is.eof()) {
      ReadToken(is, false, &token);
       if (token == "<ParamRange>") ReadBasicType(is, false, &param_range);
      else if (token == "<CellDim>") ReadBasicType(is, false, &cell_dim_);
      else if (token == "<LearnRateCoef>") ReadBasicType(is, false, &learn_rate_coef_);
      else if (token == "<BiasLearnRateCoef>") ReadBasicType(is, false, &bias_learn_rate_coef_);
      else if (token == "<CellClip>") ReadBasicType(is, false, &cell_clip_);
      else if (token == "<DiffClip>") ReadBasicType(is, false, &diff_clip_);
      else if (token == "<CellDiffClip>") ReadBasicType(is, false, &cell_diff_clip_);
      else if (token == "<GradClip>") ReadBasicType(is, false, &grad_clip_);
      else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
                     << " (ParamRange|CellDim|LearnRateCoef|BiasLearnRateCoef|CellClip|DiffClip|GradClip)";
    }

    // init the weights and biases (from uniform dist.),
    w_gifo_x_.Resize(4*cell_dim_, input_dim_, kUndefined);
    w_gifo_r_.Resize(4*cell_dim_, proj_dim_, kUndefined);
    bias_.Resize(4*cell_dim_, kUndefined);
    peephole_i_c_.Resize(cell_dim_, kUndefined);
    peephole_f_c_.Resize(cell_dim_, kUndefined);
    peephole_o_c_.Resize(cell_dim_, kUndefined);
    w_r_m_.Resize(proj_dim_, cell_dim_, kUndefined);
    //       (mean), (range)
    RandUniform(0.0, 2.0 * param_range, &w_gifo_x_);
    RandUniform(0.0, 2.0 * param_range, &w_gifo_r_);
    RandUniform(0.0, 2.0 * param_range, &bias_);
    RandUniform(0.0, 2.0 * param_range, &peephole_i_c_);
    RandUniform(0.0, 2.0 * param_range, &peephole_f_c_);
    RandUniform(0.0, 2.0 * param_range, &peephole_o_c_);
    RandUniform(0.0, 2.0 * param_range, &w_r_m_);

    KALDI_ASSERT(cell_dim_ > 0);
    KALDI_ASSERT(learn_rate_coef_ >= 0.0);
    KALDI_ASSERT(bias_learn_rate_coef_ >= 0.0);
  }

NumParams()

int32 NumParams ( ) const

inlinevirtual

Number of trainable parameters,.

Implements UpdatableComponent.

Definition at line 181 of file nnet-lstm-projected.h.

References LstmProjected::bias_, LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

Referenced by LstmProjected::GetGradient(), LstmProjected::GetParams(), and LstmProjected::SetParams().

                          {
    return ( w_gifo_x_.NumRows() * w_gifo_x_.NumCols() +
         w_gifo_r_.NumRows() * w_gifo_r_.NumCols() +
         bias_.Dim() +
         peephole_i_c_.Dim() +
         peephole_f_c_.Dim() +
         peephole_o_c_.Dim() +
         w_r_m_.NumRows() * w_r_m_.NumCols() );
  }

PropagateFnc()

void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in, CuMatrixBase< BaseFloat > *  out  )

inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 365 of file nnet-lstm-projected.h.

References CuMatrixBase< Real >::AddMatMat(), LstmProjected::bias_, LstmProjected::cell_clip_, LstmProjected::cell_dim_, CuMatrixBase< Real >::CopyFromMat(), KALDI_ASSERT, kaldi::kNoTrans, kaldi::kSetZero, kaldi::kTrans, CuMatrixBase< Real >::NumRows(), MultistreamComponent::NumStreams(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::prev_nnet_state_, LstmProjected::proj_dim_, LstmProjected::propagate_buf_, LstmProjected::ResetStreams(), CuMatrixBase< Real >::RowRange(), MultistreamComponent::sequence_lengths_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

                                                  {

    // reset context on each sentence if 'sequence_lengths_' not set
    // (happens in 'nnet-forward' or 'single-stream' training),
    if (sequence_lengths_.size() == 0) {
      ResetStreams(std::vector<int32>(1, 1));
    }

    KALDI_ASSERT(in.NumRows() % NumStreams() == 0);
    int32 T = in.NumRows() / NumStreams();
    int32 S = NumStreams();

    // buffers,
    propagate_buf_.Resize((T+2)*S, 7 * cell_dim_ + proj_dim_, kSetZero);
    if (prev_nnet_state_.NumRows() != NumStreams()) {
      prev_nnet_state_.Resize(NumStreams(), 7*cell_dim_ + 1*proj_dim_, kSetZero); // lazy init,
    } else {
      propagate_buf_.RowRange(0, S).CopyFromMat(prev_nnet_state_); // use the 'previous-state',
    }

    // split activations by neuron types,
    CuSubMatrix<BaseFloat> YG(propagate_buf_.ColRange(0*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YI(propagate_buf_.ColRange(1*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YF(propagate_buf_.ColRange(2*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YO(propagate_buf_.ColRange(3*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YC(propagate_buf_.ColRange(4*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YH(propagate_buf_.ColRange(5*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YM(propagate_buf_.ColRange(6*cell_dim_, cell_dim_));
    CuSubMatrix<BaseFloat> YR(propagate_buf_.ColRange(7*cell_dim_, proj_dim_));
    CuSubMatrix<BaseFloat> YGIFO(propagate_buf_.ColRange(0, 4*cell_dim_));

    // x -> g, i, f, o, not recurrent, do it all in once
    YGIFO.RowRange(1*S, T*S).AddMatMat(1.0, in, kNoTrans, w_gifo_x_, kTrans, 0.0);

    // bias -> g, i, f, o
    YGIFO.RowRange(1*S, T*S).AddVecToRows(1.0, bias_);

    // BufferPadding [T0]:dummy, [1, T]:current sequence, [T+1]:dummy
    for (int t = 1; t <= T; t++) {
      // multistream buffers for current time-step,
      CuSubMatrix<BaseFloat> y_all(propagate_buf_.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_g(YG.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_i(YI.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_f(YF.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_o(YO.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_c(YC.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_h(YH.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_m(YM.RowRange(t*S, S));
       CuSubMatrix<BaseFloat> y_r(YR.RowRange(t*S, S));
      CuSubMatrix<BaseFloat> y_gifo(YGIFO.RowRange(t*S, S));

      // r(t-1) -> g, i, f, o
      y_gifo.AddMatMat(1.0, YR.RowRange((t-1)*S, S), kNoTrans, w_gifo_r_, kTrans,  1.0);

      // c(t-1) -> i(t) via peephole
      y_i.AddMatDiagVec(1.0, YC.RowRange((t-1)*S, S), kNoTrans, peephole_i_c_, 1.0);

      // c(t-1) -> f(t) via peephole
      y_f.AddMatDiagVec(1.0, YC.RowRange((t-1)*S, S), kNoTrans, peephole_f_c_, 1.0);

      // i, f sigmoid squashing
      y_i.Sigmoid(y_i);
      y_f.Sigmoid(y_f);

      // g tanh squashing
      y_g.Tanh(y_g);

      // g * i -> c
      y_c.AddMatMatElements(1.0, y_g, y_i, 0.0);
      // c(t-1) * f -> c(t) via forget-gate
      y_c.AddMatMatElements(1.0, YC.RowRange((t-1)*S, S), y_f, 1.0);

      if (cell_clip_ > 0.0) {
        y_c.ApplyFloor(-cell_clip_);   // optional clipping of cell activation,
        y_c.ApplyCeiling(cell_clip_);  // google paper Interspeech2014: LSTM for LVCSR
      }

      // c(t) -> o(t) via peephole (non-recurrent, using c(t))
      y_o.AddMatDiagVec(1.0, y_c, kNoTrans, peephole_o_c_, 1.0);

      // o sigmoid squashing,
      y_o.Sigmoid(y_o);

      // h tanh squashing,
      y_h.Tanh(y_c);

      // h * o -> m via output gate,
      y_m.AddMatMatElements(1.0, y_h, y_o, 0.0);

      // m -> r
      y_r.AddMatMat(1.0, y_m, kNoTrans, w_r_m_, kTrans, 0.0);

      // set zeros to padded frames,
      if (sequence_lengths_.size() > 0) {
        for (int s = 0; s < S; s++) {
          if (t > sequence_lengths_[s]) {
            y_all.Row(s).SetZero();
          }
        }
      }
    }

    // set the 'projection layer' output as the LSTM output,
    out->CopyFromMat(YR.RowRange(1*S, T*S));

    // the state in the last 'frame' is transferred (can be zero vector)
    prev_nnet_state_.CopyFromMat(propagate_buf_.RowRange(T*S, S));
  }

ReadData()

void ReadData ( std::istream &  is, bool  binary  )

inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 107 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, kaldi::ExpectToken(), LstmProjected::grad_clip_, KALDI_ASSERT, KALDI_ERR, UpdatableComponent::learn_rate_coef_, kaldi::Peek(), kaldi::PeekToken(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, kaldi::ReadBasicType(), kaldi::ReadToken(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

                                             {
    // Read all the '<Tokens>' in arbitrary order,
    while ('<' == Peek(is, binary)) {
      std::string token;
      int first_char = PeekToken(is, binary);
      switch (first_char) {
        case 'C': ReadToken(is, false, &token);
           if (token == "<CellDim>") ReadBasicType(is, binary, &cell_dim_);
          else if (token == "<CellClip>") ReadBasicType(is, binary, &cell_clip_);
          else if (token == "<CellDiffClip>") ReadBasicType(is, binary, &cell_diff_clip_);
          else if (token == "<ClipGradient>") ReadBasicType(is, binary, &grad_clip_); // bwd-compat.
          else KALDI_ERR << "Unknown token: " << token;
          break;
        case 'L': ExpectToken(is, binary, "<LearnRateCoef>");
          ReadBasicType(is, binary, &learn_rate_coef_);
          break;
        case 'B': ExpectToken(is, binary, "<BiasLearnRateCoef>");
          ReadBasicType(is, binary, &bias_learn_rate_coef_);
          break;
        case 'D': ExpectToken(is, binary, "<DiffClip>");
          ReadBasicType(is, binary, &diff_clip_);
          break;
        case 'G': ExpectToken(is, binary, "<GradClip>");
          ReadBasicType(is, binary, &grad_clip_);
          break;
        default: ReadToken(is, false, &token);
          KALDI_ERR << "Unknown token: " << token;
      }
    }
    KALDI_ASSERT(cell_dim_ != 0);

    // Read the model parameters,
    w_gifo_x_.Read(is, binary);
    w_gifo_r_.Read(is, binary);
    bias_.Read(is, binary);

    peephole_i_c_.Read(is, binary);
    peephole_f_c_.Read(is, binary);
    peephole_o_c_.Read(is, binary);

    w_r_m_.Read(is, binary);
  }

ResetStreams()

void ResetStreams ( const std::vector< int32 > &  stream_reset_flag )

inlinevirtual

TODO: Do we really need this?

Reimplemented from MultistreamComponent.

Definition at line 352 of file nnet-lstm-projected.h.

References LstmProjected::cell_dim_, KALDI_ASSERT, kaldi::kSetZero, MultistreamComponent::NumStreams(), LstmProjected::prev_nnet_state_, and LstmProjected::proj_dim_.

Referenced by LstmProjected::PropagateFnc().

                                                               {
    KALDI_ASSERT(NumStreams() == stream_reset_flag.size());
    if (prev_nnet_state_.NumRows() != stream_reset_flag.size()) {
      prev_nnet_state_.Resize(NumStreams(), 7*cell_dim_ + 1*proj_dim_, kSetZero);
    } else {
      for (int s = 0; s < NumStreams(); s++) {
        if (stream_reset_flag[s] == 1) {
          prev_nnet_state_.Row(s).SetZero();
        }
      }
    }
  }

SetParams()

void SetParams ( const VectorBase< BaseFloat > &  params )

inlinevirtual

Set the trainable parameters from, reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 249 of file nnet-lstm-projected.h.

References LstmProjected::bias_, VectorBase< Real >::Dim(), KALDI_ASSERT, LstmProjected::NumParams(), LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, VectorBase< Real >::Range(), LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, and LstmProjected::w_r_m_.

                                                      {
    KALDI_ASSERT(params.Dim() == NumParams());
    int32 offset, len;

    offset = 0;    len = w_gifo_x_.NumRows() * w_gifo_x_.NumCols();
    w_gifo_x_.CopyRowsFromVec(params.Range(offset, len));

    offset += len; len = w_gifo_r_.NumRows() * w_gifo_r_.NumCols();
    w_gifo_r_.CopyRowsFromVec(params.Range(offset, len));

    offset += len; len = bias_.Dim();
    bias_.CopyFromVec(params.Range(offset, len));

    offset += len; len = peephole_i_c_.Dim();
    peephole_i_c_.CopyFromVec(params.Range(offset, len));

    offset += len; len = peephole_f_c_.Dim();
    peephole_f_c_.CopyFromVec(params.Range(offset, len));

    offset += len; len = peephole_o_c_.Dim();
    peephole_o_c_.CopyFromVec(params.Range(offset, len));

    offset += len; len = w_r_m_.NumRows() * w_r_m_.NumCols();
    w_r_m_.CopyRowsFromVec(params.Range(offset, len));

    offset += len;
    KALDI_ASSERT(offset == NumParams());
  }

Update()

void Update ( const CuMatrixBase< BaseFloat > &  input, const CuMatrixBase< BaseFloat > &  diff  )

inlinevirtual

Compute gradient and update parameters,.

Implements UpdatableComponent.

Definition at line 654 of file nnet-lstm-projected.h.

References LstmProjected::bias_, LstmProjected::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::grad_clip_, NnetTrainOptions::learn_rate, UpdatableComponent::learn_rate_coef_, UpdatableComponent::opts_, LstmProjected::peephole_f_c_, LstmProjected::peephole_f_c_corr_, LstmProjected::peephole_i_c_, LstmProjected::peephole_i_c_corr_, LstmProjected::peephole_o_c_, LstmProjected::peephole_o_c_corr_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_r_corr_, LstmProjected::w_gifo_x_, LstmProjected::w_gifo_x_corr_, LstmProjected::w_r_m_, and LstmProjected::w_r_m_corr_.

                                                   {

    // apply the gradient clipping,
    if (grad_clip_ > 0.0) {
      w_gifo_x_corr_.ApplyFloor(-grad_clip_);
      w_gifo_x_corr_.ApplyCeiling(grad_clip_);
      w_gifo_r_corr_.ApplyFloor(-grad_clip_);
      w_gifo_r_corr_.ApplyCeiling(grad_clip_);
      bias_corr_.ApplyFloor(-grad_clip_);
      bias_corr_.ApplyCeiling(grad_clip_);
      w_r_m_corr_.ApplyFloor(-grad_clip_);
      w_r_m_corr_.ApplyCeiling(grad_clip_);
      peephole_i_c_corr_.ApplyFloor(-grad_clip_);
      peephole_i_c_corr_.ApplyCeiling(grad_clip_);
      peephole_f_c_corr_.ApplyFloor(-grad_clip_);
      peephole_f_c_corr_.ApplyCeiling(grad_clip_);
      peephole_o_c_corr_.ApplyFloor(-grad_clip_);
      peephole_o_c_corr_.ApplyCeiling(grad_clip_);
    }

    const BaseFloat lr  = opts_.learn_rate;

    w_gifo_x_.AddMat(-lr * learn_rate_coef_, w_gifo_x_corr_);
    w_gifo_r_.AddMat(-lr * learn_rate_coef_, w_gifo_r_corr_);
    bias_.AddVec(-lr * bias_learn_rate_coef_, bias_corr_, 1.0);

    peephole_i_c_.AddVec(-lr * bias_learn_rate_coef_, peephole_i_c_corr_, 1.0);
    peephole_f_c_.AddVec(-lr * bias_learn_rate_coef_, peephole_f_c_corr_, 1.0);
    peephole_o_c_.AddVec(-lr * bias_learn_rate_coef_, peephole_o_c_corr_, 1.0);

    w_r_m_.AddMat(-lr * learn_rate_coef_, w_r_m_corr_);
  }

WriteData()

void WriteData ( std::ostream &  os, bool  binary  ) const

inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 150 of file nnet-lstm-projected.h.

References LstmProjected::bias_, UpdatableComponent::bias_learn_rate_coef_, LstmProjected::cell_clip_, LstmProjected::cell_diff_clip_, LstmProjected::cell_dim_, LstmProjected::diff_clip_, LstmProjected::grad_clip_, UpdatableComponent::learn_rate_coef_, LstmProjected::peephole_f_c_, LstmProjected::peephole_i_c_, LstmProjected::peephole_o_c_, LstmProjected::w_gifo_r_, LstmProjected::w_gifo_x_, LstmProjected::w_r_m_, kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                    {
    WriteToken(os, binary, "<CellDim>");
    WriteBasicType(os, binary, cell_dim_);

    WriteToken(os, binary, "<LearnRateCoef>");
    WriteBasicType(os, binary, learn_rate_coef_);
    WriteToken(os, binary, "<BiasLearnRateCoef>");
    WriteBasicType(os, binary, bias_learn_rate_coef_);

    WriteToken(os, binary, "<CellClip>");
    WriteBasicType(os, binary, cell_clip_);
    WriteToken(os, binary, "<DiffClip>");
    WriteBasicType(os, binary, diff_clip_);
    WriteToken(os, binary, "<CellDiffClip>");
    WriteBasicType(os, binary, cell_diff_clip_);
    WriteToken(os, binary, "<GradClip>");
    WriteBasicType(os, binary, grad_clip_);

    // write model parameters,
    if (!binary) os << "\n";
    w_gifo_x_.Write(os, binary);
    w_gifo_r_.Write(os, binary);
    bias_.Write(os, binary);

    peephole_i_c_.Write(os, binary);
    peephole_f_c_.Write(os, binary);
    peephole_o_c_.Write(os, binary);

    w_r_m_.Write(os, binary);
  }

Member Data Documentation

backpropagate_buf_

CuMatrix<BaseFloat> backpropagate_buf_

private

Definition at line 731 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), and LstmProjected::InfoGradient().

bias_

CuVector<BaseFloat> bias_

private

Definition at line 710 of file nnet-lstm-projected.h.

Referenced by LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

bias_corr_

CuVector<BaseFloat> bias_corr_

private

Definition at line 711 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

cell_clip_

BaseFloat cell_clip_

private

Clipping of 'cell-values' in forward pass (per-frame),.

Definition at line 693 of file nnet-lstm-projected.h.

Referenced by LstmProjected::Info(), LstmProjected::InfoGradient(), LstmProjected::InitData(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), and LstmProjected::WriteData().

cell_diff_clip_

BaseFloat cell_diff_clip_

private

Clipping of 'cell-derivatives' accumulated over CEC (per-frame),.

Definition at line 695 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::InitData(), LstmProjected::ReadData(), and LstmProjected::WriteData().

cell_dim_

int32 cell_dim_

private

Definition at line 690 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::Info(), LstmProjected::InfoGradient(), LstmProjected::InitData(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::ResetStreams(), and LstmProjected::WriteData().

diff_clip_

BaseFloat diff_clip_

private

Clipping of 'derivatives' in backprop (per-frame),.

Definition at line 694 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::Info(), LstmProjected::InfoGradient(), LstmProjected::InitData(), LstmProjected::ReadData(), and LstmProjected::WriteData().

grad_clip_

BaseFloat grad_clip_

private

Clipping of the updates,.

Definition at line 696 of file nnet-lstm-projected.h.

Referenced by LstmProjected::Info(), LstmProjected::InfoGradient(), LstmProjected::InitData(), LstmProjected::ReadData(), LstmProjected::Update(), and LstmProjected::WriteData().

peephole_f_c_

CuVector<BaseFloat> peephole_f_c_

private

Definition at line 716 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

peephole_f_c_corr_

CuVector<BaseFloat> peephole_f_c_corr_

private

Definition at line 720 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

peephole_i_c_

CuVector<BaseFloat> peephole_i_c_

private

Definition at line 715 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

peephole_i_c_corr_

CuVector<BaseFloat> peephole_i_c_corr_

private

Definition at line 719 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

peephole_o_c_

CuVector<BaseFloat> peephole_o_c_

private

Definition at line 717 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

peephole_o_c_corr_

CuVector<BaseFloat> peephole_o_c_corr_

private

Definition at line 721 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

prev_nnet_state_

CuMatrix<BaseFloat> prev_nnet_state_

private

Definition at line 699 of file nnet-lstm-projected.h.

Referenced by LstmProjected::PropagateFnc(), and LstmProjected::ResetStreams().

proj_dim_

int32 proj_dim_

private

recurrent projection layer dim

Definition at line 691 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::InfoGradient(), LstmProjected::InitData(), LstmProjected::PropagateFnc(), and LstmProjected::ResetStreams().

propagate_buf_

CuMatrix<BaseFloat> propagate_buf_

private

Definition at line 728 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::InfoGradient(), and LstmProjected::PropagateFnc().

w_gifo_r_

CuMatrix<BaseFloat> w_gifo_r_

private

Definition at line 706 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

w_gifo_r_corr_

CuMatrix<BaseFloat> w_gifo_r_corr_

private

Definition at line 707 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

w_gifo_x_

CuMatrix<BaseFloat> w_gifo_x_

private

Definition at line 702 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

w_gifo_x_corr_

CuMatrix<BaseFloat> w_gifo_x_corr_

private

Definition at line 703 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

w_r_m_

CuMatrix<BaseFloat> w_r_m_

private

Definition at line 724 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::GetParams(), LstmProjected::Info(), LstmProjected::InitData(), LstmProjected::NumParams(), LstmProjected::PropagateFnc(), LstmProjected::ReadData(), LstmProjected::SetParams(), LstmProjected::Update(), and LstmProjected::WriteData().

w_r_m_corr_

CuMatrix<BaseFloat> w_r_m_corr_

private

Definition at line 725 of file nnet-lstm-projected.h.

Referenced by LstmProjected::BackpropagateFnc(), LstmProjected::GetGradient(), LstmProjected::InfoGradient(), and LstmProjected::Update().

---

The documentation for this class was generated from the following file:

• nnet/nnet-lstm-projected.h