All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
ParametricRelu Class Reference

#include <nnet-parametric-relu.h>

Inheritance diagram for ParametricRelu:
Collaboration diagram for ParametricRelu:

Public Member Functions

 ParametricRelu (int32 dim_in, int32 dim_out)
 
 ~ParametricRelu ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void InitData (std::istream &is)
 Initialize the content of the component by the 'line' from the prototype,. More...
 
void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
int32 NumParams () const
 Number of trainable parameters,. More...
 
void GetGradient (VectorBase< BaseFloat > *gradient) const
 Get gradient reshaped as a vector,. More...
 
void GetParams (VectorBase< BaseFloat > *params) const
 Get the trainable parameters reshaped as a vector,. More...
 
void SetParams (const VectorBase< BaseFloat > &params)
 Set the trainable parameters from, reshaped as a vector,. More...
 
std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
void Update (const CuMatrixBase< BaseFloat > &input, const CuMatrixBase< BaseFloat > &diff)
 Compute gradient and update parameters,. More...
 
- Public Member Functions inherited from UpdatableComponent
 UpdatableComponent (int32 input_dim, int32 output_dim)
 
virtual ~UpdatableComponent ()
 
bool IsUpdatable () const
 Check if contains trainable parameters,. More...
 
virtual void SetTrainOptions (const NnetTrainOptions &opts)
 Set the training options to the component,. More...
 
const NnetTrainOptionsGetTrainOptions () const
 Get the training options from the component,. More...
 
virtual void SetLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient,. More...
 
virtual void SetBiasLearnRateCoef (BaseFloat val)
 Set the learn-rate coefficient for bias,. More...
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
virtual bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 

Private Attributes

CuVector< BaseFloatalpha_
 Vector of 'alphas', one value per neuron. More...
 
CuVector< BaseFloatbeta_
 Vector of 'betas', one value per neuron. More...
 
CuVector< BaseFloatalpha_corr_
 Vector of 'alpha' updates. More...
 
CuVector< BaseFloatbeta_corr_
 Vector of 'beta' updates. More...
 
CuMatrix< BaseFloatalpha_aux_
 Auxiliary matrix for getting 'alpha' updates,. More...
 
CuMatrix< BaseFloatbeta_aux_
 Auxiliary matrix for getting 'beta' updates,. More...
 
BaseFloat alpha_learn_rate_coef_
 Controls learning rate for alpha (0.0 disables learning),. More...
 
BaseFloat beta_learn_rate_coef_
 Controls learning rate for beta (0.0 disables learning),. More...
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 
- Protected Attributes inherited from UpdatableComponent
NnetTrainOptions opts_
 Option-class with training hyper-parameters,. More...
 
BaseFloat learn_rate_coef_
 Scalar applied to learning rate for weight matrices (to be used in ::Update method),. More...
 
BaseFloat bias_learn_rate_coef_
 Scalar applied to learning rate for bias (to be used in ::Update method),. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Detailed Description

Definition at line 34 of file nnet-parametric-relu.h.

Constructor & Destructor Documentation

ParametricRelu ( int32  dim_in,
int32  dim_out 
)
inline

Definition at line 36 of file nnet-parametric-relu.h.

Referenced by ParametricRelu::Copy().

36  :
37  UpdatableComponent(dim_in, dim_out),
38  alpha_(dim_out),
39  beta_(dim_out),
40  alpha_corr_(dim_out),
41  beta_corr_(dim_out),
44  { }
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
UpdatableComponent(int32 input_dim, int32 output_dim)
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
CuVector< BaseFloat > beta_corr_
Vector of 'beta' updates.
CuVector< BaseFloat > alpha_corr_
Vector of 'alpha' updates.
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.
~ParametricRelu ( )
inline

Definition at line 46 of file nnet-parametric-relu.h.

47  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 158 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::beta_, and CuMatrixBase< Real >::DiffParametricRelu().

161  {
162  // in_diff = (in > 0 ? alpha * out_diff : beta * out_diff)
163  in_diff->DiffParametricRelu(in, out_diff, alpha_, beta_);
164  }
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
void DiffParametricRelu(const CuMatrixBase< Real > &value, const CuMatrixBase< Real > &diff, const CuVectorBase< Real > &alpha, const CuVectorBase< Real > &beta)
Differentiate backward through the parametric relu function.
Definition: cu-matrix.cc:1350
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 49 of file nnet-parametric-relu.h.

References ParametricRelu::ParametricRelu().

49 { return new ParametricRelu(*this); }
ParametricRelu(int32 dim_in, int32 dim_out)
void GetGradient ( VectorBase< BaseFloat > *  gradient) const
inlinevirtual

Get gradient reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 113 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::alpha_corr_, ParametricRelu::beta_, ParametricRelu::beta_corr_, VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, ParametricRelu::NumParams(), and VectorBase< Real >::Range().

113  {
114  KALDI_ASSERT(gradient->Dim() == NumParams());
115  int32 alpha_num_elem = alpha_.Dim();
116  int32 beta_num_elem = beta_.Dim();
117  gradient->Range(0, alpha_num_elem).CopyFromVec(Vector<BaseFloat>(alpha_corr_));
118  gradient->Range(alpha_num_elem, beta_num_elem).CopyFromVec(Vector<BaseFloat>(beta_corr_));
119  }
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuVector< BaseFloat > beta_corr_
Vector of 'beta' updates.
int32 NumParams() const
Number of trainable parameters,.
CuVector< BaseFloat > alpha_corr_
Vector of 'alpha' updates.
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void GetParams ( VectorBase< BaseFloat > *  params) const
inlinevirtual

Get the trainable parameters reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 121 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::beta_, VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, ParametricRelu::NumParams(), and VectorBase< Real >::Range().

121  {
122  KALDI_ASSERT(params->Dim() == NumParams());
123  int32 alpha_num_elem = alpha_.Dim();
124  int32 beta_num_elem = beta_.Dim();
125  params->Range(0, alpha_num_elem).CopyFromVec(Vector<BaseFloat>(alpha_));
126  params->Range(alpha_num_elem, beta_num_elem).CopyFromVec(Vector<BaseFloat>(beta_));
127  }
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumParams() const
Number of trainable parameters,.
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 50 of file nnet-parametric-relu.h.

References Component::kParametricRelu.

std::string Info ( ) const
inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 137 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::alpha_learn_rate_coef_, ParametricRelu::beta_, ParametricRelu::beta_learn_rate_coef_, kaldi::nnet1::MomentStatistics(), and kaldi::nnet1::ToString().

137  {
138  return std::string("\n alpha") +
140  ", alpha-lr-coef " + ToString(alpha_learn_rate_coef_) +
141  "\n beta" + MomentStatistics(beta_) +
142  ", beta-lr-coef " + ToString(beta_learn_rate_coef_);
143  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.
std::string InfoGradient ( ) const
inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 144 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_corr_, ParametricRelu::alpha_learn_rate_coef_, ParametricRelu::beta_corr_, ParametricRelu::beta_learn_rate_coef_, kaldi::nnet1::MomentStatistics(), and kaldi::nnet1::ToString().

144  {
145  return std::string("\n alpha_grad") +
147  ", alpha-lr-coef " + ToString(alpha_learn_rate_coef_) +
148  "\n beta_grad" + MomentStatistics(beta_corr_) +
149  ", beta-lr-coef " + ToString(beta_learn_rate_coef_);
150  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
std::string MomentStatistics(const VectorBase< Real > &vec)
Get a string with statistics of the data in a vector, so we can print them easily.
Definition: nnet-utils.h:63
CuVector< BaseFloat > beta_corr_
Vector of 'beta' updates.
CuVector< BaseFloat > alpha_corr_
Vector of 'alpha' updates.
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.
void InitData ( std::istream &  is)
inlinevirtual

Initialize the content of the component by the 'line' from the prototype,.

Implements UpdatableComponent.

Definition at line 52 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::alpha_learn_rate_coef_, ParametricRelu::beta_, ParametricRelu::beta_learn_rate_coef_, KALDI_ERR, kaldi::ReadBasicType(), kaldi::ReadToken(), and CuVectorBase< Real >::Set().

52  {
53  // define options
54  BaseFloat alpha = 1.0, beta = 0.0;
55 
56  // parse config
57  std::string token;
58  while (is >> std::ws, !is.eof()) {
59  ReadToken(is, false, &token);
60  if (token == "<Alpha>") ReadBasicType(is, false, &alpha);
61  else if (token == "<Beta>") ReadBasicType(is, false, &beta);
62  else if (token == "<AlphaLearnRateCoef>") ReadBasicType(is, false, &alpha_learn_rate_coef_);
63  else if (token == "<BetaLearnRateCoef>") ReadBasicType(is, false, &beta_learn_rate_coef_);
64  else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
65  << " (Alpha|Beta|AlphaLearnRateCoef|BetaLearnRateCoef)";
66  }
67 
68  // Initialize trainable parameters,
69  alpha_.Set(alpha);
70  beta_.Set(beta);
71  }
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void Set(Real value)
Definition: cu-vector.cc:1024
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ERR
Definition: kaldi-error.h:127
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.
int32 NumParams ( ) const
inlinevirtual

Number of trainable parameters,.

Implements UpdatableComponent.

Definition at line 109 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::beta_, and CuVectorBase< Real >::Dim().

Referenced by ParametricRelu::GetGradient(), ParametricRelu::GetParams(), and ParametricRelu::SetParams().

109  {
110  return alpha_.Dim() + beta_.Dim();
111  }
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 152 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::beta_, and CuMatrixBase< Real >::ParametricRelu().

153  {
154  // out = (in < 0.0 ? aplha*in : beta*in)
155  out->ParametricRelu(in, alpha_, beta_);
156  }
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
void ParametricRelu(const CuMatrixBase< Real > &src, const CuVectorBase< Real > &alpha, const CuVectorBase< Real > &beta)
Compute the parametric rectified linear unit function; element by element, *this = src * (src > 0 ...
Definition: cu-matrix.cc:1316
void ReadData ( std::istream &  is,
bool  binary 
)
inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 73 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::alpha_learn_rate_coef_, ParametricRelu::beta_, ParametricRelu::beta_learn_rate_coef_, CuVectorBase< Real >::Dim(), kaldi::ExpectToken(), KALDI_ASSERT, KALDI_ERR, Component::output_dim_, kaldi::Peek(), kaldi::PeekToken(), CuVector< Real >::Read(), kaldi::ReadBasicType(), and kaldi::ReadToken().

73  {
74  // Read all the '<Tokens>' in arbitrary order,
75  while ('<' == Peek(is, binary)) {
76  int first_char = PeekToken(is, binary);
77  switch (first_char) {
78  case 'A': ExpectToken(is, binary, "<AlphaLearnRateCoef>");
80  break;
81  case 'B': ExpectToken(is, binary, "<BetaLearnRateCoef>");
83  break;
84  default:
85  std::string token;
86  ReadToken(is, false, &token);
87  KALDI_ERR << "Unknown token: " << token;
88  }
89  }
90  // ParametricRelu scaling parameters
91  alpha_.Read(is, binary);
92  beta_.Read(is, binary);
95  }
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void Read(std::istream &is, bool binary)
I/O.
Definition: cu-vector.cc:862
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
#define KALDI_ERR
Definition: kaldi-error.h:127
int PeekToken(std::istream &is, bool binary)
PeekToken will return the first character of the next token, or -1 if end of file.
Definition: io-funcs.cc:170
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
int32 output_dim_
Dimension of the output of the Component,.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.
void SetParams ( const VectorBase< BaseFloat > &  params)
inlinevirtual

Set the trainable parameters from, reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 129 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::beta_, CuVectorBase< Real >::CopyFromVec(), VectorBase< Real >::Dim(), CuVectorBase< Real >::Dim(), KALDI_ASSERT, ParametricRelu::NumParams(), and VectorBase< Real >::Range().

129  {
130  KALDI_ASSERT(params.Dim() == NumParams());
131  int32 alpha_num_elem = alpha_.Dim();
132  int32 beta_num_elem = beta_.Dim();
133  alpha_.CopyFromVec(params.Range(0, alpha_num_elem));
134  beta_.CopyFromVec(params.Range(alpha_num_elem, beta_num_elem));
135  }
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
void CopyFromVec(const CuVectorBase< Real > &src)
Copy functions; these will crash if the dimension do not match.
Definition: cu-vector.cc:970
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumParams() const
Number of trainable parameters,.
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void Update ( const CuMatrixBase< BaseFloat > &  input,
const CuMatrixBase< BaseFloat > &  diff 
)
inlinevirtual

Compute gradient and update parameters,.

Implements UpdatableComponent.

Definition at line 166 of file nnet-parametric-relu.h.

References CuVectorBase< Real >::AddRowSumMat(), CuVectorBase< Real >::AddVec(), ParametricRelu::alpha_, ParametricRelu::alpha_aux_, ParametricRelu::alpha_corr_, ParametricRelu::alpha_learn_rate_coef_, CuMatrixBase< Real >::ApplyCeiling(), CuMatrixBase< Real >::ApplyFloor(), ParametricRelu::beta_, ParametricRelu::beta_aux_, ParametricRelu::beta_corr_, ParametricRelu::beta_learn_rate_coef_, NnetTrainOptions::learn_rate, NnetTrainOptions::momentum, CuMatrixBase< Real >::MulElements(), and UpdatableComponent::opts_.

167  {
168  // we use these hyperparameters,
171  const BaseFloat mmt = opts_.momentum;
172 
173  if (alpha_learn_rate_coef_ > 0.0) {
174  // get gradient,
175  alpha_aux_ = input;
176  alpha_aux_.ApplyFloor(0.0); // masking positive Relu inputs,
177  alpha_aux_.MulElements(diff);
179  // update,
180  alpha_.AddVec(-alpha_lr, alpha_corr_);
181  }
182  if (beta_learn_rate_coef_ > 0.0) {
183  // get gradient,
184  beta_aux_ = input;
185  beta_aux_.ApplyCeiling(0.0); // masking positive Relu inputs,
186  beta_aux_.MulElements(diff);
187  beta_corr_.AddRowSumMat(1.0, beta_aux_, mmt);
188  beta_.AddVec(-beta_lr, beta_corr_);
189  }
190  }
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
NnetTrainOptions opts_
Option-class with training hyper-parameters,.
void ApplyCeiling(Real ceiling_val)
Definition: cu-matrix.cc:2383
CuMatrix< BaseFloat > alpha_aux_
Auxiliary matrix for getting 'alpha' updates,.
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
void ApplyFloor(Real floor_val)
Definition: cu-matrix.cc:2365
float BaseFloat
Definition: kaldi-types.h:29
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
void MulElements(const CuMatrixBase< Real > &A)
Multiply two matrices elementwise: C = C .* A.
Definition: cu-matrix.cc:652
CuVector< BaseFloat > beta_corr_
Vector of 'beta' updates.
CuMatrix< BaseFloat > beta_aux_
Auxiliary matrix for getting 'beta' updates,.
CuVector< BaseFloat > alpha_corr_
Vector of 'alpha' updates.
void AddVec(Real alpha, const CuVectorBase< Real > &vec, Real beta=1.0)
Definition: cu-vector.cc:1126
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.
void WriteData ( std::ostream &  os,
bool  binary 
) const
inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 97 of file nnet-parametric-relu.h.

References ParametricRelu::alpha_, ParametricRelu::alpha_learn_rate_coef_, ParametricRelu::beta_, ParametricRelu::beta_learn_rate_coef_, CuVector< Real >::Write(), kaldi::WriteBasicType(), and kaldi::WriteToken().

97  {
98  WriteToken(os, binary, "<AlphaLearnRateCoef>");
100  WriteToken(os, binary, "<BetaLearnRateCoef>");
102 
103  // ParametricRelu scales for each neuron,
104  if (!binary) os << "\n";
105  alpha_.Write(os, binary);
106  beta_.Write(os, binary);
107  }
BaseFloat beta_learn_rate_coef_
Controls learning rate for beta (0.0 disables learning),.
CuVector< BaseFloat > alpha_
Vector of 'alphas', one value per neuron.
void Write(std::ostream &is, bool binary) const
Definition: cu-vector.cc:872
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
CuVector< BaseFloat > beta_
Vector of 'betas', one value per neuron.
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
BaseFloat alpha_learn_rate_coef_
Controls learning rate for alpha (0.0 disables learning),.

Member Data Documentation

CuMatrix<BaseFloat> alpha_aux_
private

Auxiliary matrix for getting 'alpha' updates,.

Definition at line 200 of file nnet-parametric-relu.h.

Referenced by ParametricRelu::Update().

CuVector<BaseFloat> alpha_corr_
private

Vector of 'alpha' updates.

Definition at line 196 of file nnet-parametric-relu.h.

Referenced by ParametricRelu::GetGradient(), ParametricRelu::InfoGradient(), and ParametricRelu::Update().

BaseFloat alpha_learn_rate_coef_
private

Controls learning rate for alpha (0.0 disables learning),.

Definition at line 205 of file nnet-parametric-relu.h.

Referenced by ParametricRelu::Info(), ParametricRelu::InfoGradient(), ParametricRelu::InitData(), ParametricRelu::ReadData(), ParametricRelu::Update(), and ParametricRelu::WriteData().

CuMatrix<BaseFloat> beta_aux_
private

Auxiliary matrix for getting 'beta' updates,.

Definition at line 202 of file nnet-parametric-relu.h.

Referenced by ParametricRelu::Update().

CuVector<BaseFloat> beta_corr_
private
BaseFloat beta_learn_rate_coef_
private

The documentation for this class was generated from the following file: