All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
SimpleSentenceAveragingComponent Class Reference

SimpleSentenceAveragingComponent does not have nested network, it is intended to be used inside of a <ParallelComponent>. More...

#include <nnet-sentence-averaging-component.h>

Inheritance diagram for SimpleSentenceAveragingComponent:
Collaboration diagram for SimpleSentenceAveragingComponent:

Public Member Functions

 SimpleSentenceAveragingComponent (int32 dim_in, int32 dim_out)
 
 ~SimpleSentenceAveragingComponent ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void InitData (std::istream &is)
 Virtual interface for initialization and I/O,. More...
 
void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
virtual bool IsUpdatable () const
 Check if componeny has 'Updatable' interface (trainable components),. More...
 
virtual bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 

Private Attributes

CuVector< BaseFloataverage_row_
 Auxiliary buffer for forward propagation (for average vector),. More...
 
CuVector< BaseFloataverage_diff_
 Auxiliary buffer for back-propagation (for average vector),. More...
 
BaseFloat gradient_boost_
 Scalar applied on gradient in backpropagation,. More...
 
BaseFloat shrinkage_
 Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences),. More...
 
bool only_summing_
 Removes normalization term from arithmetic mean (when true). More...
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Detailed Description

SimpleSentenceAveragingComponent does not have nested network, it is intended to be used inside of a <ParallelComponent>.

For training use 'nnet-train-perutt'.

The sentence-averaging typically leads to small gradients, so we boost it 100x by default (boost = multiply, it's equivalent to applying learning-rate factor).

Definition at line 42 of file nnet-sentence-averaging-component.h.

Constructor & Destructor Documentation

SimpleSentenceAveragingComponent ( int32  dim_in,
int32  dim_out 
)
inline

Definition at line 44 of file nnet-sentence-averaging-component.h.

Referenced by SimpleSentenceAveragingComponent::Copy().

44  :
45  Component(dim_in, dim_out),
46  gradient_boost_(100.0),
47  shrinkage_(0.0),
48  only_summing_(false)
49  { }
Component(int32 input_dim, int32 output_dim)
Generic interface of a component,.
bool only_summing_
Removes normalization term from arithmetic mean (when true).
BaseFloat gradient_boost_
Scalar applied on gradient in backpropagation,.
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...

Definition at line 51 of file nnet-sentence-averaging-component.h.

52  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 137 of file nnet-sentence-averaging-component.h.

References CuVectorBase< Real >::AddRowSumMat(), CuMatrixBase< Real >::AddVecToRows(), SimpleSentenceAveragingComponent::average_diff_, SimpleSentenceAveragingComponent::gradient_boost_, CuMatrixBase< Real >::NumRows(), SimpleSentenceAveragingComponent::only_summing_, Component::OutputDim(), CuVector< Real >::Resize(), and SimpleSentenceAveragingComponent::shrinkage_.

140  {
141  // When averaging, a single frame from input influenced all the frames
142  // on the output. So the derivative w.r.t. single input frame is a sum
143  // of the output derivatives, scaled by the averaging constant 1/K.
144  //
145  // In the same time all the input frames of the average influenced
146  // all the output frames. So the loss derivarive is same for all
147  // the input frames coming to the averaging.
148  //
149  // getting the average output diff,
151  if (only_summing_) {
152  average_diff_.AddRowSumMat(1.0, out_diff, 0.0);
153  } else {
154  average_diff_.AddRowSumMat(1.0/(out_diff.NumRows()+shrinkage_), out_diff, 0.0);
155  }
156  // copy the derivative into the input diff, (applying gradient-boost!!)
158  }
CuVector< BaseFloat > average_diff_
Auxiliary buffer for back-propagation (for average vector),.
void AddVecToRows(Real alpha, const CuVectorBase< Real > &row, Real beta=1.0)
(for each row r of *this), r = alpha * row + beta * r
Definition: cu-matrix.cc:1111
int32 OutputDim() const
Get the dimension of the output,.
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
bool only_summing_
Removes normalization term from arithmetic mean (when true).
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
BaseFloat gradient_boost_
Scalar applied on gradient in backpropagation,.
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 54 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::SimpleSentenceAveragingComponent().

54  {
55  return new SimpleSentenceAveragingComponent(*this);
56  }
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 58 of file nnet-sentence-averaging-component.h.

References Component::kSimpleSentenceAveragingComponent.

std::string Info ( ) const
inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 115 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::gradient_boost_, SimpleSentenceAveragingComponent::only_summing_, SimpleSentenceAveragingComponent::shrinkage_, and kaldi::nnet1::ToString().

Referenced by SimpleSentenceAveragingComponent::InfoGradient().

115  {
116  return std::string("\n gradient-boost ") + ToString(gradient_boost_) +
117  ", shrinkage: " + ToString(shrinkage_) +
118  ", only summing: " + ToString(only_summing_);
119  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
bool only_summing_
Removes normalization term from arithmetic mean (when true).
BaseFloat gradient_boost_
Scalar applied on gradient in backpropagation,.
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...
std::string InfoGradient ( ) const
inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 120 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::Info().

120  {
121  return Info();
122  }
std::string Info() const
Print some additional info (after and the dims),.
void InitData ( std::istream &  is)
inlinevirtual

Virtual interface for initialization and I/O,.

Initialize internal data of a component

Reimplemented from Component.

Definition at line 62 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::gradient_boost_, KALDI_ERR, SimpleSentenceAveragingComponent::only_summing_, kaldi::ReadBasicType(), kaldi::ReadToken(), and SimpleSentenceAveragingComponent::shrinkage_.

62  {
63  // parse config
64  std::string token;
65  while (is >> std::ws, !is.eof()) {
66  ReadToken(is, false, &token);
67  if (token == "<GradientBoost>") ReadBasicType(is, false, &gradient_boost_);
68  else if (token == "<Shrinkage>") ReadBasicType(is, false, &shrinkage_);
69  else if (token == "<OnlySumming>") ReadBasicType(is, false, &only_summing_);
70  else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
71  << " (GradientBoost|Shrinkage|OnlySumming)";
72  }
73  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
bool only_summing_
Removes normalization term from arithmetic mean (when true).
#define KALDI_ERR
Definition: kaldi-error.h:127
BaseFloat gradient_boost_
Scalar applied on gradient in backpropagation,.
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...
void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 124 of file nnet-sentence-averaging-component.h.

References CuVectorBase< Real >::AddRowSumMat(), CuMatrixBase< Real >::AddVecToRows(), SimpleSentenceAveragingComponent::average_row_, Component::InputDim(), CuMatrixBase< Real >::NumRows(), SimpleSentenceAveragingComponent::only_summing_, CuVector< Real >::Resize(), and SimpleSentenceAveragingComponent::shrinkage_.

125  {
126  // get the average row-vector,
128  if (only_summing_) {
129  average_row_.AddRowSumMat(1.0, in, 0.0);
130  } else {
131  average_row_.AddRowSumMat(1.0/(in.NumRows()+shrinkage_), in, 0.0);
132  }
133  // copy it on the output,
134  out->AddVecToRows(1.0, average_row_, 0.0);
135  }
void AddVecToRows(Real alpha, const CuVectorBase< Real > &row, Real beta=1.0)
(for each row r of *this), r = alpha * row + beta * r
Definition: cu-matrix.cc:1111
CuVector< BaseFloat > average_row_
Auxiliary buffer for forward propagation (for average vector),.
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
bool only_summing_
Removes normalization term from arithmetic mean (when true).
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
int32 InputDim() const
Get the dimension of the input,.
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...
void ReadData ( std::istream &  is,
bool  binary 
)
inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 75 of file nnet-sentence-averaging-component.h.

References kaldi::ExpectToken(), SimpleSentenceAveragingComponent::gradient_boost_, KALDI_WARN, SimpleSentenceAveragingComponent::only_summing_, kaldi::Peek(), kaldi::PeekToken(), kaldi::ReadBasicType(), and SimpleSentenceAveragingComponent::shrinkage_.

75  {
76  bool end_loop = false;
77  while (!end_loop && '<' == Peek(is, binary)) {
78  int first_char = PeekToken(is, binary);
79  switch (first_char) {
80  case 'G': ExpectToken(is, binary, "<GradientBoost>");
81  ReadBasicType(is, binary, &gradient_boost_);
82  break;
83  case 'S': ExpectToken(is, binary, "<Shrinkage>");
84  ReadBasicType(is, binary, &shrinkage_);
85  break;
86  case 'O': ExpectToken(is, binary, "<OnlySumming>");
87  // compatibility trick,
88  // in some models 'only_summing_' was float '0.0',
89  // from now 'only_summing_' is 'bool':
90  try {
91  ReadBasicType(is, binary, &only_summing_);
92  } catch(const std::exception &e) {
93  KALDI_WARN << "ERROR was handled by exception!";
94  BaseFloat dummy_float;
95  ReadBasicType(is, binary, &dummy_float);
96  }
97  break;
98  case '!':
99  ExpectToken(is, binary, "<!EndOfComponent>");
100  default:
101  end_loop = true;
102  }
103  }
104  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
bool only_summing_
Removes normalization term from arithmetic mean (when true).
float BaseFloat
Definition: kaldi-types.h:29
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
#define KALDI_WARN
Definition: kaldi-error.h:130
int PeekToken(std::istream &is, bool binary)
PeekToken will return the first character of the next token, or -1 if end of file.
Definition: io-funcs.cc:170
BaseFloat gradient_boost_
Scalar applied on gradient in backpropagation,.
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...
void WriteData ( std::ostream &  os,
bool  binary 
) const
inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 106 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::gradient_boost_, SimpleSentenceAveragingComponent::only_summing_, SimpleSentenceAveragingComponent::shrinkage_, kaldi::WriteBasicType(), and kaldi::WriteToken().

106  {
107  WriteToken(os, binary, "<GradientBoost>");
108  WriteBasicType(os, binary, gradient_boost_);
109  WriteToken(os, binary, "<Shrinkage>");
110  WriteBasicType(os, binary, shrinkage_);
111  WriteToken(os, binary, "<OnlySumming>");
112  WriteBasicType(os, binary, only_summing_);
113  }
bool only_summing_
Removes normalization term from arithmetic mean (when true).
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
BaseFloat gradient_boost_
Scalar applied on gradient in backpropagation,.
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
BaseFloat shrinkage_
Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences)...

Member Data Documentation

CuVector<BaseFloat> average_diff_
private

Auxiliary buffer for back-propagation (for average vector),.

Definition at line 165 of file nnet-sentence-averaging-component.h.

Referenced by SimpleSentenceAveragingComponent::BackpropagateFnc().

CuVector<BaseFloat> average_row_
private

Auxiliary buffer for forward propagation (for average vector),.

Definition at line 162 of file nnet-sentence-averaging-component.h.

Referenced by SimpleSentenceAveragingComponent::PropagateFnc().


The documentation for this class was generated from the following file: