All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
Rescale Class Reference

Rescale the data column-wise by a vector (can be used for global variance normalization) More...

#include <nnet-various.h>

Inheritance diagram for Rescale:
Collaboration diagram for Rescale:

Public Member Functions

 Rescale (int32 dim_in, int32 dim_out)
 
 ~Rescale ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void InitData (std::istream &is)
 Initialize the content of the component by the 'line' from the prototype,. More...
 
void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
int32 NumParams () const
 Number of trainable parameters,. More...
 
void GetGradient (VectorBase< BaseFloat > *gradient) const
 Get gradient reshaped as a vector,. More...
 
void GetParams (VectorBase< BaseFloat > *params) const
 Get the trainable parameters reshaped as a vector,. More...
 
void SetParams (const VectorBase< BaseFloat > &params)
 Set the trainable parameters from, reshaped as a vector,. More...
 
std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
void Update (const CuMatrixBase< BaseFloat > &input, const CuMatrixBase< BaseFloat > &diff)
 Compute gradient and update parameters,. More...
 
void SetLearnRateCoef (BaseFloat c)
 Set the learn-rate coefficient,. More...
 
- Public Member Functions inherited from UpdatableComponent
 UpdatableComponent (int32 input_dim, int32 output_dim)
 
virtual ~UpdatableComponent ()
 
bool IsUpdatable () const
 Check if contains trainable parameters,. More...
 
virtual void SetTrainOptions (const NnetTrainOptions &opts)
 Set the training options to the component,. More...
 
const NnetTrainOptionsGetTrainOptions () const
 Get the training options from the component,. More...
 
virtual void SetBiasLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient for bias,. More...
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
virtual bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 

Protected Attributes

CuVector< BaseFloatscale_data_
 
CuVector< BaseFloatscale_data_grad_
 
- Protected Attributes inherited from UpdatableComponent
NnetTrainOptions opts_
 Option-class with training hyper-parameters,. More...
 
BaseFloat learn_rate_coef_
 Scalar applied to learning rate for weight matrices (to be used in ::Update method),. More...
 
BaseFloat bias_learn_rate_coef_
 Scalar applied to learning rate for bias (to be used in ::Update method),. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 

Detailed Description

Rescale the data column-wise by a vector (can be used for global variance normalization)

Definition at line 404 of file nnet-various.h.

Constructor & Destructor Documentation

Rescale ( int32  dim_in,
int32  dim_out 
)
inline

Definition at line 406 of file nnet-various.h.

Referenced by Rescale::Copy().

406  :
407  UpdatableComponent(dim_in, dim_out),
408  scale_data_(dim_in)
409  { }
UpdatableComponent(int32 input_dim, int32 output_dim)
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
~Rescale ( )
inline

Definition at line 411 of file nnet-various.h.

412  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 486 of file nnet-various.h.

References CuMatrixBase< Real >::CopyFromMat(), CuMatrixBase< Real >::MulColsVec(), and Rescale::scale_data_.

489  {
490  // derivatives are scaled with the scale_data_,
491  in_diff->CopyFromMat(out_diff);
492  in_diff->MulColsVec(scale_data_);
493  }
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
void MulColsVec(const CuVectorBase< Real > &scale)
scale i'th column by scale[i]
Definition: cu-matrix.cc:750
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 414 of file nnet-various.h.

References Rescale::Rescale().

414 { return new Rescale(*this); }
Rescale(int32 dim_in, int32 dim_out)
Definition: nnet-various.h:406
void GetGradient ( VectorBase< BaseFloat > *  gradient) const
inlinevirtual

Get gradient reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 452 of file nnet-various.h.

References CuVectorBase< Real >::CopyToVec(), VectorBase< Real >::Dim(), KALDI_ASSERT, Rescale::NumParams(), and Rescale::scale_data_grad_.

452  {
453  KALDI_ASSERT(gradient->Dim() == NumParams());
454  scale_data_grad_.CopyToVec(gradient);
455  }
CuVector< BaseFloat > scale_data_grad_
Definition: nnet-various.h:512
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void CopyToVec(VectorBase< OtherReal > *dst) const
Definition: cu-vector.cc:839
int32 NumParams() const
Number of trainable parameters,.
Definition: nnet-various.h:450
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
void GetParams ( VectorBase< BaseFloat > *  params) const
inlinevirtual

Get the trainable parameters reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 457 of file nnet-various.h.

References CuVectorBase< Real >::CopyToVec(), VectorBase< Real >::Dim(), KALDI_ASSERT, Rescale::NumParams(), and Rescale::scale_data_.

Referenced by kaldi::ConvertRescaleComponent().

457  {
458  KALDI_ASSERT(params->Dim() == NumParams());
459  scale_data_.CopyToVec(params);
460  }
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void CopyToVec(VectorBase< OtherReal > *dst) const
Definition: cu-vector.cc:839
int32 NumParams() const
Number of trainable parameters,.
Definition: nnet-various.h:450
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 415 of file nnet-various.h.

References Component::kRescale.

std::string Info ( ) const
inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 467 of file nnet-various.h.

References UpdatableComponent::learn_rate_coef_, kaldi::nnet1::MomentStatistics(), Rescale::scale_data_, and kaldi::nnet1::ToString().

467  {
468  return std::string("\n scale_data") +
470  ", lr-coef " + ToString(learn_rate_coef_);
471  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
std::string InfoGradient ( ) const
inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 473 of file nnet-various.h.

References UpdatableComponent::learn_rate_coef_, kaldi::nnet1::MomentStatistics(), Rescale::scale_data_grad_, and kaldi::nnet1::ToString().

473  {
474  return std::string("\n scale_data_grad") +
476  ", lr-coef " + ToString(learn_rate_coef_);
477  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
CuVector< BaseFloat > scale_data_grad_
Definition: nnet-various.h:512
void InitData ( std::istream &  is)
inlinevirtual

Initialize the content of the component by the 'line' from the prototype,.

Implements UpdatableComponent.

Definition at line 417 of file nnet-various.h.

References Component::InputDim(), KALDI_ERR, kaldi::kSetZero, UpdatableComponent::learn_rate_coef_, kaldi::ReadBasicType(), kaldi::ReadToken(), CuVector< Real >::Resize(), Rescale::scale_data_, and CuVectorBase< Real >::Set().

417  {
418  // define options
419  float init_param = 0.0;
420  // parse config
421  std::string token;
422  while (is >> std::ws, !is.eof()) {
423  ReadToken(is, false, &token);
424  if (token == "<InitParam>") ReadBasicType(is, false, &init_param);
425  else if (token == "<LearnRateCoef>") ReadBasicType(is, false, &learn_rate_coef_);
426  else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
427  << " (InitParam)";
428  }
429  // initialize
431  scale_data_.Set(init_param);
432  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Set(Real value)
Definition: cu-vector.cc:1024
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
#define KALDI_ERR
Definition: kaldi-error.h:127
int32 InputDim() const
Get the dimension of the input,.
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
int32 NumParams ( ) const
inlinevirtual

Number of trainable parameters,.

Implements UpdatableComponent.

Definition at line 450 of file nnet-various.h.

References CuVectorBase< Real >::Dim(), and Rescale::scale_data_.

Referenced by kaldi::ConvertRescaleComponent(), Rescale::GetGradient(), Rescale::GetParams(), and Rescale::SetParams().

450 { return scale_data_.Dim(); }
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 479 of file nnet-various.h.

References CuMatrixBase< Real >::CopyFromMat(), CuMatrixBase< Real >::MulColsVec(), and Rescale::scale_data_.

480  {
481  // copy, rescale the data,
482  out->CopyFromMat(in);
483  out->MulColsVec(scale_data_);
484  }
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
void MulColsVec(const CuVectorBase< Real > &scale)
scale i'th column by scale[i]
Definition: cu-matrix.cc:750
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
void ReadData ( std::istream &  is,
bool  binary 
)
inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 434 of file nnet-various.h.

References kaldi::ExpectToken(), UpdatableComponent::learn_rate_coef_, kaldi::Peek(), CuVector< Real >::Read(), kaldi::ReadBasicType(), and Rescale::scale_data_.

434  {
435  // optional learning-rate coef,
436  if ('<' == Peek(is, binary)) {
437  ExpectToken(is, binary, "<LearnRateCoef>");
438  ReadBasicType(is, binary, &learn_rate_coef_);
439  }
440  // read the shift data
441  scale_data_.Read(is, binary);
442  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Read(std::istream &is, bool binary)
I/O.
Definition: cu-vector.cc:862
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
void SetLearnRateCoef ( BaseFloat  val)
inlinevirtual

Set the learn-rate coefficient,.

Reimplemented from UpdatableComponent.

Definition at line 508 of file nnet-various.h.

References UpdatableComponent::learn_rate_coef_.

508 { learn_rate_coef_ = c; }
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void SetParams ( const VectorBase< BaseFloat > &  params)
inlinevirtual

Set the trainable parameters from, reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 462 of file nnet-various.h.

References CuVectorBase< Real >::CopyFromVec(), VectorBase< Real >::Dim(), KALDI_ASSERT, Rescale::NumParams(), and Rescale::scale_data_.

Referenced by main().

462  {
463  KALDI_ASSERT(params.Dim() == NumParams());
464  scale_data_.CopyFromVec(params);
465  }
void CopyFromVec(const CuVectorBase< Real > &src)
Copy functions; these will crash if the dimension do not match.
Definition: cu-vector.cc:970
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumParams() const
Number of trainable parameters,.
Definition: nnet-various.h:450
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
void Update ( const CuMatrixBase< BaseFloat > &  input,
const CuMatrixBase< BaseFloat > &  diff 
)
inlinevirtual

Compute gradient and update parameters,.

Implements UpdatableComponent.

Definition at line 495 of file nnet-various.h.

References CuVectorBase< Real >::AddRowSumMat(), CuVectorBase< Real >::AddVec(), Component::InputDim(), kaldi::kSetZero, NnetTrainOptions::learn_rate, UpdatableComponent::learn_rate_coef_, CuMatrixBase< Real >::MulElements(), UpdatableComponent::opts_, CuVector< Real >::Resize(), Rescale::scale_data_, and Rescale::scale_data_grad_.

496  {
497  // we use following hyperparameters from the option class,
498  const BaseFloat lr = opts_.learn_rate;
499  // gradient,
501  CuMatrix<BaseFloat> gradient_aux(diff);
502  gradient_aux.MulElements(input);
503  scale_data_grad_.AddRowSumMat(1.0, gradient_aux, 0.0);
504  // update,
506  }
NnetTrainOptions opts_
Option-class with training hyper-parameters,.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
float BaseFloat
Definition: kaldi-types.h:29
CuVector< BaseFloat > scale_data_grad_
Definition: nnet-various.h:512
int32 InputDim() const
Get the dimension of the input,.
void AddVec(Real alpha, const CuVectorBase< Real > &vec, Real beta=1.0)
Definition: cu-vector.cc:1126
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511
void WriteData ( std::ostream &  os,
bool  binary 
) const
inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 444 of file nnet-various.h.

References UpdatableComponent::learn_rate_coef_, Rescale::scale_data_, CuVector< Real >::Write(), kaldi::WriteBasicType(), and kaldi::WriteToken().

444  {
445  WriteToken(os, binary, "<LearnRateCoef>");
446  WriteBasicType(os, binary, learn_rate_coef_);
447  scale_data_.Write(os, binary);
448  }
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Write(std::ostream &is, bool binary) const
Definition: cu-vector.cc:872
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
CuVector< BaseFloat > scale_data_
Definition: nnet-various.h:511

Member Data Documentation

CuVector<BaseFloat> scale_data_grad_
protected

Definition at line 512 of file nnet-various.h.

Referenced by Rescale::GetGradient(), Rescale::InfoGradient(), and Rescale::Update().


The documentation for this class was generated from the following file: