All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
Dropout Class Reference

#include <nnet-activation.h>

Inheritance diagram for Dropout:
Collaboration diagram for Dropout:

Public Member Functions

 Dropout (int32 dim_in, int32 dim_out)
 
 ~Dropout ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void InitData (std::istream &is)
 Virtual interface for initialization and I/O,. More...
 
void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
BaseFloat GetDropoutRate ()
 
void SetDropoutRate (BaseFloat dr)
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
virtual bool IsUpdatable () const
 Check if componeny has 'Updatable' interface (trainable components),. More...
 
virtual bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 
virtual std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 

Private Attributes

BaseFloat dropout_rate_
 probability that a neuron is dropped, More...
 
CuRand< BaseFloatrand_
 generator of random numbers, More...
 
CuMatrix< BaseFloatdropout_mask_
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Detailed Description

Definition at line 269 of file nnet-activation.h.

Constructor & Destructor Documentation

Dropout ( int32  dim_in,
int32  dim_out 
)
inline

Definition at line 271 of file nnet-activation.h.

Referenced by Dropout::Copy().

271  :
272  Component(dim_in, dim_out),
273  dropout_rate_(0.5)
274  { }
Component(int32 input_dim, int32 output_dim)
Generic interface of a component,.
BaseFloat dropout_rate_
probability that a neuron is dropped,
~Dropout ( )
inline

Definition at line 276 of file nnet-activation.h.

277  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 342 of file nnet-activation.h.

References CuMatrixBase< Real >::CopyFromMat(), Dropout::dropout_mask_, Dropout::dropout_rate_, CuMatrixBase< Real >::MulElements(), and CuMatrixBase< Real >::Scale().

345  {
346  in_diff->CopyFromMat(out_diff);
347  // use same mask on the error derivatives...
348  in_diff->MulElements(dropout_mask_);
349  // enlarge the output to fit same dynamic range as w/o dropout
350  in_diff->Scale(1.0 / (1.0 - dropout_rate_));
351  }
void Scale(Real value)
Definition: cu-matrix.cc:608
CuMatrix< BaseFloat > dropout_mask_
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
BaseFloat dropout_rate_
probability that a neuron is dropped,
void MulElements(const CuMatrixBase< Real > &A)
Multiply two matrices elementwise: C = C .* A.
Definition: cu-matrix.cc:652
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 279 of file nnet-activation.h.

References Dropout::Dropout().

279 { return new Dropout(*this); }
Dropout(int32 dim_in, int32 dim_out)
BaseFloat GetDropoutRate ( )
inline

Definition at line 353 of file nnet-activation.h.

References Dropout::dropout_rate_.

Referenced by Nnet::SetDropoutRate().

353 { return dropout_rate_; }
BaseFloat dropout_rate_
probability that a neuron is dropped,
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 280 of file nnet-activation.h.

References Component::kDropout.

std::string Info ( ) const
inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 325 of file nnet-activation.h.

References Dropout::dropout_rate_, and kaldi::nnet1::ToString().

325  {
326  return std::string("<DropoutRate> ") + ToString(dropout_rate_);
327  }
std::string ToString(const T &t)
Convert basic type to a string (please don't overuse),.
Definition: nnet-utils.h:52
BaseFloat dropout_rate_
probability that a neuron is dropped,
void InitData ( std::istream &  is)
inlinevirtual

Virtual interface for initialization and I/O,.

Initialize internal data of a component

Reimplemented from Component.

Definition at line 282 of file nnet-activation.h.

References Dropout::dropout_rate_, KALDI_ASSERT, KALDI_ERR, kaldi::ReadBasicType(), and kaldi::ReadToken().

282  {
283  is >> std::ws; // eat-up whitespace
284  // parse config
285  std::string token;
286  while (is >> std::ws, !is.eof()) {
287  ReadToken(is, false, &token);
288  if (token == "<DropoutRate>") ReadBasicType(is, false, &dropout_rate_);
289  else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
290  << " (DropoutRate)";
291  }
292  KALDI_ASSERT(dropout_rate_ >= 0.0 && dropout_rate_ < 1.0);
293  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
BaseFloat dropout_rate_
probability that a neuron is dropped,
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 329 of file nnet-activation.h.

References CuMatrixBase< Real >::Add(), CuMatrixBase< Real >::CopyFromMat(), Dropout::dropout_mask_, Dropout::dropout_rate_, CuMatrixBase< Real >::Heaviside(), CuMatrixBase< Real >::MulElements(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), Dropout::rand_, CuRand< Real >::RandUniform(), CuMatrix< Real >::Resize(), and CuMatrixBase< Real >::Scale().

330  {
331  out->CopyFromMat(in);
332  // set N inputs to zero, according to the 'dropout_rate_' ...
333  dropout_mask_.Resize(out->NumRows(), out->NumCols());
334  rand_.RandUniform(&dropout_mask_); // [0..1]
335  dropout_mask_.Add(-dropout_rate_); // [(-rate)..(1-rate)]
336  dropout_mask_.Heaviside(dropout_mask_); // (x > 0.0 ? 1 : 0)
338  // rescale to keep the same dynamic range as w/o dropout,
339  out->Scale(1.0 / (1.0 - dropout_rate_));
340  }
void Heaviside(const CuMatrixBase< Real > &src)
Set each element to the Heaviside function of the corresponding element of "src", which we define as ...
Definition: cu-matrix.cc:2325
void Scale(Real value)
Definition: cu-matrix.cc:608
CuRand< BaseFloat > rand_
generator of random numbers,
CuMatrix< BaseFloat > dropout_mask_
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
void RandUniform(CuMatrixBase< Real > *tgt)
Fill with uniform [0..1] floats,.
Definition: cu-rand.cc:60
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
BaseFloat dropout_rate_
probability that a neuron is dropped,
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void MulElements(const CuMatrixBase< Real > &A)
Multiply two matrices elementwise: C = C .* A.
Definition: cu-matrix.cc:652
void Add(Real value)
Definition: cu-matrix.cc:546
void ReadData ( std::istream &  is,
bool  binary 
)
inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 295 of file nnet-activation.h.

References Dropout::dropout_rate_, kaldi::ExpectToken(), KALDI_ASSERT, KALDI_ERR, kaldi::Peek(), kaldi::PeekToken(), kaldi::ReadBasicType(), and kaldi::ReadToken().

295  {
296  // Read all the '<Tokens>' in arbitrary order,
297  bool finished = false;
298  while ('<' == Peek(is, binary) && !finished) {
299  std::string token;
300  int first_char = PeekToken(is, binary);
301  switch (first_char) {
302  case 'D': ReadToken(is, false, &token);
303  if (token == "<DropoutRate>") ReadBasicType(is, binary, &dropout_rate_);
304  else if (token == "<DropoutRetention>") { /* compatibility */
305  BaseFloat dropout_retention;
306  ReadBasicType(is, binary, &dropout_retention);
307  dropout_rate_ = 1.0 - dropout_retention;
308  } else KALDI_ERR << "Unknown token: " << token;
309  break;
310  case '!': ExpectToken(is, binary, "<!EndOfComponent>");
311  finished = true;
312  break;
313  default: ReadToken(is, false, &token);
314  KALDI_ERR << "Unknown token: " << token;
315  }
316  }
317  KALDI_ASSERT(dropout_rate_ >= 0.0 && dropout_rate_ < 1.0);
318  }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
int Peek(std::istream &is, bool binary)
Peek consumes whitespace (if binary == false) and then returns the peek() value of the stream...
Definition: io-funcs.cc:145
float BaseFloat
Definition: kaldi-types.h:29
BaseFloat dropout_rate_
probability that a neuron is dropped,
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
#define KALDI_ERR
Definition: kaldi-error.h:127
int PeekToken(std::istream &is, bool binary)
PeekToken will return the first character of the next token, or -1 if end of file.
Definition: io-funcs.cc:170
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SetDropoutRate ( BaseFloat  dr)
inline

Definition at line 355 of file nnet-activation.h.

References Dropout::dropout_rate_, and KALDI_ASSERT.

Referenced by Nnet::SetDropoutRate().

355  {
356  dropout_rate_ = dr;
357  KALDI_ASSERT(dropout_rate_ >= 0.0 && dropout_rate_ < 1.0);
358  }
BaseFloat dropout_rate_
probability that a neuron is dropped,
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void WriteData ( std::ostream &  os,
bool  binary 
) const
inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 320 of file nnet-activation.h.

References Dropout::dropout_rate_, kaldi::WriteBasicType(), and kaldi::WriteToken().

320  {
321  WriteToken(os, binary, "<DropoutRate>");
322  WriteBasicType(os, binary, dropout_rate_);
323  }
BaseFloat dropout_rate_
probability that a neuron is dropped,
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34

Member Data Documentation

CuMatrix<BaseFloat> dropout_mask_
private

Definition at line 365 of file nnet-activation.h.

Referenced by Dropout::BackpropagateFnc(), and Dropout::PropagateFnc().

CuRand<BaseFloat> rand_
private

generator of random numbers,

Definition at line 363 of file nnet-activation.h.

Referenced by Dropout::PropagateFnc().


The documentation for this class was generated from the following file: