All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
AffineTransform Class Reference

#include <nnet-affine-transform.h>

Inheritance diagram for AffineTransform:
Collaboration diagram for AffineTransform:

Public Member Functions

 AffineTransform (int32 dim_in, int32 dim_out)
 
 ~AffineTransform ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void InitData (std::istream &is)
 Initialize the content of the component by the 'line' from the prototype,. More...
 
void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
int32 NumParams () const
 Number of trainable parameters,. More...
 
void GetGradient (VectorBase< BaseFloat > *gradient) const
 Get gradient reshaped as a vector,. More...
 
void GetParams (VectorBase< BaseFloat > *params) const
 Get the trainable parameters reshaped as a vector,. More...
 
void SetParams (const VectorBase< BaseFloat > &params)
 Set the trainable parameters from, reshaped as a vector,. More...
 
std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
void Update (const CuMatrixBase< BaseFloat > &input, const CuMatrixBase< BaseFloat > &diff)
 Compute gradient and update parameters,. More...
 
const CuVectorBase< BaseFloat > & GetBias () const
 Accessors to the component parameters,. More...
 
void SetBias (const CuVectorBase< BaseFloat > &bias)
 
const CuMatrixBase< BaseFloat > & GetLinearity () const
 
void SetLinearity (const CuMatrixBase< BaseFloat > &linearity)
 
- Public Member Functions inherited from UpdatableComponent
 UpdatableComponent (int32 input_dim, int32 output_dim)
 
virtual ~UpdatableComponent ()
 
bool IsUpdatable () const
 Check if contains trainable parameters,. More...
 
virtual void SetTrainOptions (const NnetTrainOptions &opts)
 Set the training options to the component,. More...
 
const NnetTrainOptionsGetTrainOptions () const
 Get the training options from the component,. More...
 
virtual void SetLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient,. More...
 
virtual void SetBiasLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient for bias,. More...
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
virtual bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 

Private Attributes

CuMatrix< BaseFloatlinearity_
 
CuVector< BaseFloatbias_
 
CuMatrix< BaseFloatlinearity_corr_
 
CuVector< BaseFloatbias_corr_
 
BaseFloat max_norm_
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 
- Protected Attributes inherited from UpdatableComponent
NnetTrainOptions opts_
 Option-class with training hyper-parameters,. More...
 
BaseFloat learn_rate_coef_
 Scalar applied to learning rate for weight matrices (to be used in ::Update method),. More...
 
BaseFloat bias_learn_rate_coef_
 Scalar applied to learning rate for bias (to be used in ::Update method),. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Detailed Description

Definition at line 33 of file nnet-affine-transform.h.

Constructor & Destructor Documentation

AffineTransform ( int32  dim_in,
int32  dim_out 
)
inline

Definition at line 35 of file nnet-affine-transform.h.

Referenced by AffineTransform::Copy().

35  :
36  UpdatableComponent(dim_in, dim_out),
37  linearity_(dim_out, dim_in), bias_(dim_out),
38  linearity_corr_(dim_out, dim_in), bias_corr_(dim_out),
39  max_norm_(0.0)
40  { }
CuMatrix< BaseFloat > linearity_corr_
UpdatableComponent(int32 input_dim, int32 output_dim)
~AffineTransform ( )
inline

Definition at line 41 of file nnet-affine-transform.h.

42  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 170 of file nnet-affine-transform.h.

References CuMatrixBase< Real >::AddMatMat(), kaldi::kNoTrans, and AffineTransform::linearity_.

173  {
174  // multiply error derivative by weights
175  in_diff->AddMatMat(1.0, out_diff, kNoTrans, linearity_, kNoTrans, 0.0);
176  }
void AddMatMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType transA, const CuMatrixBase< Real > &B, MatrixTransposeType transB, Real beta)
C = alpha * A(^T)*B(^T) + beta * C.
Definition: cu-matrix.cc:1102
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 44 of file nnet-affine-transform.h.

References AffineTransform::AffineTransform().

44 { return new AffineTransform(*this); }
AffineTransform(int32 dim_in, int32 dim_out)
const CuVectorBase<BaseFloat>& GetBias ( ) const
inline

Accessors to the component parameters,.

Definition at line 219 of file nnet-affine-transform.h.

References AffineTransform::bias_.

Referenced by kaldi::ConvertAffineTransformComponent().

219 { return bias_; }
void GetGradient ( VectorBase< BaseFloat > *  gradient) const
inlinevirtual

Get gradient reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 124 of file nnet-affine-transform.h.

References AffineTransform::bias_, AffineTransform::bias_corr_, VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, AffineTransform::linearity_, AffineTransform::linearity_corr_, CuMatrixBase< Real >::NumCols(), AffineTransform::NumParams(), CuMatrixBase< Real >::NumRows(), and VectorBase< Real >::Range().

124  {
125  KALDI_ASSERT(gradient->Dim() == NumParams());
126  int32 linearity_num_elem = linearity_.NumRows() * linearity_.NumCols();
127  gradient->Range(0, linearity_num_elem).CopyRowsFromMat(linearity_corr_);
128  gradient->Range(linearity_num_elem, bias_.Dim()).CopyFromVec(bias_corr_);
129  }
CuMatrix< BaseFloat > linearity_corr_
int32 NumParams() const
Number of trainable parameters,.
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
const CuMatrixBase<BaseFloat>& GetLinearity ( ) const
inline

Definition at line 226 of file nnet-affine-transform.h.

References AffineTransform::linearity_.

Referenced by kaldi::ConvertAffineTransformComponent().

226 { return linearity_; }
void GetParams ( VectorBase< BaseFloat > *  params) const
inlinevirtual

Get the trainable parameters reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 131 of file nnet-affine-transform.h.

References AffineTransform::bias_, VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, AffineTransform::linearity_, CuMatrixBase< Real >::NumCols(), AffineTransform::NumParams(), CuMatrixBase< Real >::NumRows(), and VectorBase< Real >::Range().

131  {
132  KALDI_ASSERT(params->Dim() == NumParams());
133  int32 linearity_num_elem = linearity_.NumRows() * linearity_.NumCols();
134  params->Range(0, linearity_num_elem).CopyRowsFromMat(linearity_);
135  params->Range(linearity_num_elem, bias_.Dim()).CopyFromVec(bias_);
136  }
int32 NumParams() const
Number of trainable parameters,.
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 45 of file nnet-affine-transform.h.

References Component::kAffineTransform.

std::string Info ( ) const
inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 145 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, kaldi::nnet1::MomentStatistics(), and kaldi::nnet1::ToString().

145  {
146  return std::string("\n linearity") +
148  ", lr-coef " + ToString(learn_rate_coef_) +
149  ", max-norm " + ToString(max_norm_) +
150  "\n bias" + MomentStatistics(bias_) +
151  ", lr-coef " + ToString(bias_learn_rate_coef_);
152  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
std::string InfoGradient ( ) const
inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 153 of file nnet-affine-transform.h.

References AffineTransform::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_corr_, AffineTransform::max_norm_, kaldi::nnet1::MomentStatistics(), and kaldi::nnet1::ToString().

153  {
154  return std::string("\n linearity_grad") +
156  ", lr-coef " + ToString(learn_rate_coef_) +
157  ", max-norm " + ToString(max_norm_) +
158  "\n bias_grad" + MomentStatistics(bias_corr_) +
159  ", lr-coef " + ToString(bias_learn_rate_coef_);
160  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
CuMatrix< BaseFloat > linearity_corr_
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void InitData ( std::istream &  is)
inlinevirtual

Initialize the content of the component by the 'line' from the prototype,.

Implements UpdatableComponent.

Definition at line 47 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, Component::InputDim(), KALDI_ERR, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, Component::OutputDim(), kaldi::nnet1::RandGauss(), kaldi::nnet1::RandUniform(), kaldi::ReadBasicType(), kaldi::ReadToken(), CuVector< Real >::Resize(), and CuMatrix< Real >::Resize().

47  {
48  // define options
49  float bias_mean = -2.0, bias_range = 2.0, param_stddev = 0.1;
50  // parse config
51  std::string token;
52  while (is >> std::ws, !is.eof()) {
53  ReadToken(is, false, &token);
54  if (token == "<ParamStddev>") ReadBasicType(is, false, &param_stddev);
55  else if (token == "<BiasMean>") ReadBasicType(is, false, &bias_mean);
56  else if (token == "<BiasRange>") ReadBasicType(is, false, &bias_range);
57  else if (token == "<LearnRateCoef>") ReadBasicType(is, false, &learn_rate_coef_);
58  else if (token == "<BiasLearnRateCoef>") ReadBasicType(is, false, &bias_learn_rate_coef_);
59  else if (token == "<MaxNorm>") ReadBasicType(is, false, &max_norm_);
60  else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
61  << " (ParamStddev|BiasMean|BiasRange|LearnRateCoef|BiasLearnRateCoef)";
62  }
63 
64  //
65  // Initialize trainable parameters,
66  //
67  // Gaussian with given std_dev (mean = 0),
69  RandGauss(0.0, param_stddev, &linearity_);
70  // Uniform,
72  RandUniform(bias_mean, bias_range, &bias_);
73  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void RandUniform(BaseFloat mu, BaseFloat range, CuMatrixBase< Real > *mat, struct RandomState *state=NULL)
Fill CuMatrix with random numbers (Uniform distribution): mu = the mean value, range = the 'width' of...
Definition: nnet-utils.h:188
int32 OutputDim() const
Get the dimension of the output,.
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
#define KALDI_ERR
Definition: kaldi-error.h:127
void RandGauss(BaseFloat mu, BaseFloat sigma, CuMatrixBase< Real > *mat, struct RandomState *state=NULL)
Fill CuMatrix with random numbers (Gaussian distribution): mu = the mean value, sigma = standard devi...
Definition: nnet-utils.h:164
int32 InputDim() const
Get the dimension of the input,.
int32 NumParams ( ) const
inlinevirtual

Number of trainable parameters,.

Implements UpdatableComponent.

Definition at line 120 of file nnet-affine-transform.h.

References AffineTransform::bias_, CuVectorBase< Real >::Dim(), AffineTransform::linearity_, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

Referenced by AffineTransform::GetGradient(), AffineTransform::GetParams(), and AffineTransform::SetParams().

120  {
121  return linearity_.NumRows()*linearity_.NumCols() + bias_.Dim();
122  }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 162 of file nnet-affine-transform.h.

References CuMatrixBase< Real >::AddMatMat(), CuMatrixBase< Real >::AddVecToRows(), AffineTransform::bias_, kaldi::kNoTrans, kaldi::kTrans, and AffineTransform::linearity_.

163  {
164  // precopy bias
165  out->AddVecToRows(1.0, bias_, 0.0);
166  // multiply by weights^t
167  out->AddMatMat(1.0, in, kNoTrans, linearity_, kTrans, 1.0);
168  }
void AddVecToRows(Real alpha, const CuVectorBase< Real > &row, Real beta=1.0)
(for each row r of *this), r = alpha * row + beta * r
Definition: cu-matrix.cc:1072
void AddMatMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType transA, const CuMatrixBase< Real > &B, MatrixTransposeType transB, Real beta)
C = alpha * A(^T)*B(^T) + beta * C.
Definition: cu-matrix.cc:1102
void ReadData ( std::istream &  is,
bool  binary 
)
inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 75 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, CuVectorBase< Real >::Dim(), kaldi::ExpectToken(), Component::input_dim_, KALDI_ASSERT, KALDI_ERR, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), Component::output_dim_, kaldi::Peek(), kaldi::PeekToken(), CuVector< Real >::Read(), CuMatrix< Real >::Read(), kaldi::ReadBasicType(), and kaldi::ReadToken().

75  {
76  // Read all the '<Tokens>' in arbitrary order,
77  while ('<' == Peek(is, binary)) {
78  int first_char = PeekToken(is, binary);
79  switch (first_char) {
80  case 'L': ExpectToken(is, binary, "<LearnRateCoef>");
81  ReadBasicType(is, binary, &learn_rate_coef_);
82  break;
83  case 'B': ExpectToken(is, binary, "<BiasLearnRateCoef>");
85  break;
86  case 'M': ExpectToken(is, binary, "<MaxNorm>");
87  ReadBasicType(is, binary, &max_norm_);
88  break;
89  default:
90  std::string token;
91  ReadToken(is, false, &token);
92  KALDI_ERR << "Unknown token: " << token;
93  }
94  }
95  // Read the data (data follow the tokens),
96 
97  // weight matrix,
98  linearity_.Read(is, binary);
99  // bias vector,
100  bias_.Read(is, binary);
101 
105  }
int32 input_dim_
Data members,.
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Read(std::istream &is, bool binary)
I/O.
Definition: cu-vector.cc:862
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
#define KALDI_ERR
Definition: kaldi-error.h:127
int PeekToken(std::istream &is, bool binary)
PeekToken will return the first character of the next token, or -1 if end of file.
Definition: io-funcs.cc:170
void Read(std::istream &is, bool binary)
I/O functions.
Definition: cu-matrix.cc:459
int32 output_dim_
Dimension of the output of the Component,.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SetBias ( const CuVectorBase< BaseFloat > &  bias)
inline

Definition at line 221 of file nnet-affine-transform.h.

References AffineTransform::bias_, CuVectorBase< Real >::CopyFromVec(), CuVectorBase< Real >::Dim(), and KALDI_ASSERT.

221  {
222  KALDI_ASSERT(bias.Dim() == bias_.Dim());
223  bias_.CopyFromVec(bias);
224  }
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
void CopyFromVec(const CuVectorBase< Real > &src)
Copy functions; these will crash if the dimension do not match.
Definition: cu-vector.cc:970
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SetLinearity ( const CuMatrixBase< BaseFloat > &  linearity)
inline

Definition at line 228 of file nnet-affine-transform.h.

References CuMatrixBase< Real >::CopyFromMat(), KALDI_ASSERT, AffineTransform::linearity_, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

Referenced by MultiBasisComponent::InitData().

228  {
229  KALDI_ASSERT(linearity.NumRows() == linearity_.NumRows());
230  KALDI_ASSERT(linearity.NumCols() == linearity_.NumCols());
231  linearity_.CopyFromMat(linearity);
232  }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SetParams ( const VectorBase< BaseFloat > &  params)
inlinevirtual

Set the trainable parameters from, reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 138 of file nnet-affine-transform.h.

References AffineTransform::bias_, CuVectorBase< Real >::CopyFromVec(), CuMatrixBase< Real >::CopyRowsFromVec(), VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, AffineTransform::linearity_, CuMatrixBase< Real >::NumCols(), AffineTransform::NumParams(), CuMatrixBase< Real >::NumRows(), and VectorBase< Real >::Range().

138  {
139  KALDI_ASSERT(params.Dim() == NumParams());
140  int32 linearity_num_elem = linearity_.NumRows() * linearity_.NumCols();
141  linearity_.CopyRowsFromVec(params.Range(0, linearity_num_elem));
142  bias_.CopyFromVec(params.Range(linearity_num_elem, bias_.Dim()));
143  }
void CopyRowsFromVec(const CuVectorBase< Real > &v)
This function has two modes of operation.
Definition: cu-matrix.cc:2096
int32 NumParams() const
Number of trainable parameters,.
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void CopyFromVec(const CuVectorBase< Real > &src)
Copy functions; these will crash if the dimension do not match.
Definition: cu-vector.cc:970
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void Update ( const CuMatrixBase< BaseFloat > &  input,
const CuMatrixBase< BaseFloat > &  diff 
)
inlinevirtual

Compute gradient and update parameters,.

Implements UpdatableComponent.

Definition at line 179 of file nnet-affine-transform.h.

References CuVectorBase< Real >::AddColSumMat(), CuMatrixBase< Real >::AddMat(), CuMatrixBase< Real >::AddMatMat(), CuVectorBase< Real >::AddRowSumMat(), CuVectorBase< Real >::AddVec(), CuVectorBase< Real >::ApplyFloor(), CuVectorBase< Real >::ApplyPow(), AffineTransform::bias_, AffineTransform::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, CuVectorBase< Real >::InvertElements(), kaldi::kNoTrans, kaldi::kTrans, NnetTrainOptions::l1_penalty, NnetTrainOptions::l2_penalty, NnetTrainOptions::learn_rate, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::linearity_corr_, AffineTransform::max_norm_, NnetTrainOptions::momentum, CuMatrixBase< Real >::MulElements(), CuMatrixBase< Real >::MulRowsVec(), CuMatrixBase< Real >::NumRows(), UpdatableComponent::opts_, Component::OutputDim(), kaldi::cu::RegularizeL1(), and CuVectorBase< Real >::Scale().

180  {
181  // we use following hyperparameters from the option class
184  const BaseFloat mmt = opts_.momentum;
185  const BaseFloat l2 = opts_.l2_penalty;
186  const BaseFloat l1 = opts_.l1_penalty;
187  // we will also need the number of frames in the mini-batch
188  const int32 num_frames = input.NumRows();
189  // compute gradient (incl. momentum)
190  linearity_corr_.AddMatMat(1.0, diff, kTrans, input, kNoTrans, mmt);
191  bias_corr_.AddRowSumMat(1.0, diff, mmt);
192  // l2 regularization
193  if (l2 != 0.0) {
194  linearity_.AddMat(-lr*l2*num_frames, linearity_);
195  }
196  // l1 regularization
197  if (l1 != 0.0) {
198  cu::RegularizeL1(&linearity_, &linearity_corr_, lr*l1*num_frames, lr);
199  }
200  // update
202  bias_.AddVec(-lr_bias, bias_corr_);
203  // max-norm
204  if (max_norm_ > 0.0) {
205  CuMatrix<BaseFloat> lin_sqr(linearity_);
206  lin_sqr.MulElements(linearity_);
207  CuVector<BaseFloat> l2(OutputDim());
208  l2.AddColSumMat(1.0, lin_sqr, 0.0);
209  l2.ApplyPow(0.5); // we have per-neuron L2 norms,
210  CuVector<BaseFloat> scl(l2);
211  scl.Scale(1.0/max_norm_);
212  scl.ApplyFloor(1.0);
213  scl.InvertElements();
214  linearity_.MulRowsVec(scl); // shink to sphere!
215  }
216  }
CuMatrix< BaseFloat > linearity_corr_
NnetTrainOptions opts_
Option-class with training hyper-parameters,.
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
int32 OutputDim() const
Get the dimension of the output,.
void MulRowsVec(const CuVectorBase< Real > &scale)
scale i'th row by scale[i]
Definition: cu-matrix.cc:777
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
float BaseFloat
Definition: kaldi-types.h:29
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void AddMatMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType transA, const CuMatrixBase< Real > &B, MatrixTransposeType transB, Real beta)
C = alpha * A(^T)*B(^T) + beta * C.
Definition: cu-matrix.cc:1102
void AddMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType trans=kNoTrans)
*this += alpha * A
Definition: cu-matrix.cc:938
void AddVec(Real alpha, const CuVectorBase< Real > &vec, Real beta=1.0)
Definition: cu-vector.cc:1126
void RegularizeL1(CuMatrixBase< Real > *weight, CuMatrixBase< Real > *grad, Real l1, Real lr)
RegularizeL1 is a gradient step with l1 regularization added to the gradient.
Definition: cu-math.cc:37
void WriteData ( std::ostream &  os,
bool  binary 
) const
inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 107 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, CuVector< Real >::Write(), CuMatrixBase< Real >::Write(), kaldi::WriteBasicType(), and kaldi::WriteToken().

107  {
108  WriteToken(os, binary, "<LearnRateCoef>");
109  WriteBasicType(os, binary, learn_rate_coef_);
110  WriteToken(os, binary, "<BiasLearnRateCoef>");
112  WriteToken(os, binary, "<MaxNorm>");
113  WriteBasicType(os, binary, max_norm_);
114  if (!binary) os << "\n";
115  // weights
116  linearity_.Write(os, binary);
117  bias_.Write(os, binary);
118  }
BaseFloat bias_learn_rate_coef_
Scalar applied to learning rate for bias (to be used in ::Update method),.
BaseFloat learn_rate_coef_
Scalar applied to learning rate for weight matrices (to be used in ::Update method),.
void Write(std::ostream &is, bool binary) const
Definition: cu-vector.cc:872
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
void Write(std::ostream &os, bool binary) const
Definition: cu-matrix.cc:467

Member Data Documentation


The documentation for this class was generated from the following file: