All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
DropoutComponent Class Reference

This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two. More...

#include <nnet-component.h>

Inheritance diagram for DropoutComponent:
Collaboration diagram for DropoutComponent:

Public Member Functions

void Init (int32 dim, BaseFloat dropout_proportion=0.5, BaseFloat dropout_scale=0.0)
 dropout-proportion is the proportion that is dropped out, e.g. More...
 
 DropoutComponent (int32 dim, BaseFloat dp=0.5, BaseFloat sc=0.0)
 
 DropoutComponent ()
 
virtual int32 InputDim () const
 Get size of input vectors. More...
 
virtual int32 OutputDim () const
 Get size of output vectors. More...
 
virtual void InitFromString (std::string args)
 Initialize, typically from a line of a config file. More...
 
virtual void Read (std::istream &is, bool binary)
 
virtual void Write (std::ostream &os, bool binary) const
 Write component to stream. More...
 
virtual std::string Type () const
 
void SetDropoutScale (BaseFloat scale)
 
virtual bool BackpropNeedsInput () const
 
virtual bool BackpropNeedsOutput () const
 
virtual ComponentCopy () const
 Copy component (deep copy). More...
 
virtual void Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const
 Perform forward pass propagation Input->Output. More...
 
virtual void Backprop (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component *to_update, CuMatrix< BaseFloat > *in_deriv) const
 Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise). More...
 
virtual std::string Info () const
 
- Public Member Functions inherited from RandomComponent
void ResetGenerator ()
 
- Public Member Functions inherited from Component
 Component ()
 
virtual int32 Index () const
 Returns the index in the sequence of layers in the neural net; intended only to be used in debugging information. More...
 
virtual void SetIndex (int32 index)
 
virtual std::vector< int32 > Context () const
 Return a vector describing the temporal context this component requires for each frame of output, as a sorted list. More...
 
void Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out) const
 A non-virtual propagate function that first resizes output if necessary. More...
 
virtual ~Component ()
 

Private Attributes

int32 dim_
 
BaseFloat dropout_proportion_
 
BaseFloat dropout_scale_
 

Additional Inherited Members

- Static Public Member Functions inherited from Component
static ComponentReadNew (std::istream &is, bool binary)
 Read component from stream. More...
 
static ComponentNewFromString (const std::string &initializer_line)
 Initialize the Component from one line that will contain first the type, e.g. More...
 
static ComponentNewComponentOfType (const std::string &type)
 Return a new Component of the given type e.g. More...
 
- Protected Attributes inherited from RandomComponent
CuRand< BaseFloatrandom_generator_
 

Detailed Description

This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two.

Typically you would use this in training but not in test or when computing validation-set objective functions.

Definition at line 1584 of file nnet-component.h.

Constructor & Destructor Documentation

DropoutComponent ( int32  dim,
BaseFloat  dp = 0.5,
BaseFloat  sc = 0.0 
)
inline

Definition at line 1595 of file nnet-component.h.

References DropoutComponent::Init().

1595  {
1596  Init(dim, dp, sc);
1597  }
void Init(int32 dim, BaseFloat dropout_proportion=0.5, BaseFloat dropout_scale=0.0)
dropout-proportion is the proportion that is dropped out, e.g.
DropoutComponent ( )
inline

Definition at line 1598 of file nnet-component.h.

Referenced by DropoutComponent::Copy().

Member Function Documentation

void Backprop ( const ChunkInfo in_info,
const ChunkInfo out_info,
const CuMatrixBase< BaseFloat > &  in_value,
const CuMatrixBase< BaseFloat > &  out_value,
const CuMatrixBase< BaseFloat > &  out_deriv,
Component to_update,
CuMatrix< BaseFloat > *  in_deriv 
) const
virtual

Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise).

Note: in_value and out_value are the values of the input and output of the component, and these may be dummy variables if respectively BackpropNeedsInput() or BackpropNeedsOutput() return false for that component (not all components need these).

num_chunks lets us treat the input matrix as contiguous-in-time chunks of equal size; it only matters if splicing is involved.

Implements Component.

Definition at line 3566 of file nnet-component.cc.

References KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), CuMatrix< Real >::Resize(), kaldi::SameDim(), and CuMatrixBase< Real >::SetMatMatDivMat().

3572  {
3573  KALDI_ASSERT(SameDim(in_value, out_value) && SameDim(in_value, out_deriv));
3574  in_deriv->Resize(out_deriv.NumRows(), out_deriv.NumCols());
3575  in_deriv->SetMatMatDivMat(out_deriv, out_value, in_value);
3576 }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void SetMatMatDivMat(const CuMatrixBase< Real > &A, const CuMatrixBase< Real > &B, const CuMatrixBase< Real > &C)
*this = a * b / c (by element; when c = 0, *this = a) *this can be an alias of a, b or c safely and g...
Definition: cu-matrix.cc:1057
bool SameDim(const MatrixBase< Real > &M, const MatrixBase< Real > &N)
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
virtual bool BackpropNeedsInput ( ) const
inlinevirtual

Reimplemented from Component.

Definition at line 1610 of file nnet-component.h.

1610 { return true; }
virtual bool BackpropNeedsOutput ( ) const
inlinevirtual

Reimplemented from Component.

Definition at line 1611 of file nnet-component.h.

1611 { return true; }
std::string Info ( ) const
virtual

Reimplemented from Component.

Definition at line 3479 of file nnet-component.cc.

References DropoutComponent::dropout_proportion_, DropoutComponent::dropout_scale_, and Component::Info().

3479  {
3480  std::stringstream stream;
3481  stream << Component::Info() << ", dropout_proportion = "
3482  << dropout_proportion_ << ", dropout_scale = "
3483  << dropout_scale_;
3484  return stream.str();
3485 }
virtual std::string Info() const
void Init ( int32  dim,
BaseFloat  dropout_proportion = 0.5,
BaseFloat  dropout_scale = 0.0 
)

dropout-proportion is the proportion that is dropped out, e.g.

if 0.1, we set 10% to a low value. [note, in some older code it was interpreted as the value not dropped out, so be careful.] The low scale-value is equal to dropout_scale. The high scale-value is chosen such that the expected scale-value is one.

Definition at line 3523 of file nnet-component.cc.

References DropoutComponent::dim_, DropoutComponent::dropout_proportion_, and DropoutComponent::dropout_scale_.

Referenced by DropoutComponent::DropoutComponent(), and DropoutComponent::InitFromString().

3525  {
3526  dim_ = dim;
3527  dropout_proportion_ = dropout_proportion;
3528  dropout_scale_ = dropout_scale;
3529 }
void InitFromString ( std::string  args)
virtual

Initialize, typically from a line of a config file.

The "args" will contain any parameters that need to be passed to the Component, e.g. dimensions.

Implements Component.

Definition at line 3487 of file nnet-component.cc.

References DropoutComponent::Init(), KALDI_ERR, and kaldi::nnet2::ParseFromString().

Referenced by kaldi::nnet2::UnitTestDropoutComponent().

3487  {
3488  std::string orig_args(args);
3489  int32 dim;
3490  BaseFloat dropout_proportion = 0.5, dropout_scale = 0.0;
3491  bool ok = ParseFromString("dim", &args, &dim);
3492  ParseFromString("dropout-proportion", &args, &dropout_proportion);
3493  ParseFromString("dropout-scale", &args, &dropout_scale);
3494 
3495  if (!ok || !args.empty() || dim <= 0)
3496  KALDI_ERR << "Invalid initializer for layer of type DropoutComponent: \""
3497  << orig_args << "\"";
3498  Init(dim, dropout_proportion, dropout_scale);
3499 }
bool ParseFromString(const std::string &name, std::string *string, int32 *param)
Functions used in Init routines.
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ERR
Definition: kaldi-error.h:127
void Init(int32 dim, BaseFloat dropout_proportion=0.5, BaseFloat dropout_scale=0.0)
dropout-proportion is the proportion that is dropped out, e.g.
virtual int32 InputDim ( ) const
inlinevirtual

Get size of input vectors.

Implements Component.

Definition at line 1599 of file nnet-component.h.

References DropoutComponent::dim_.

Referenced by DropoutComponent::Propagate().

1599 { return dim_; }
virtual int32 OutputDim ( ) const
inlinevirtual

Get size of output vectors.

Implements Component.

Definition at line 1600 of file nnet-component.h.

References DropoutComponent::dim_.

1600 { return dim_; }
void Propagate ( const ChunkInfo in_info,
const ChunkInfo out_info,
const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
) const
virtual

Perform forward pass propagation Input->Output.

Each row is one frame or training example. Interpreted as "num_chunks" equally sized chunks of frames; this only matters for layers that do things like context splicing. Typically this variable will either be 1 (when we're processing a single contiguous chunk of data) or will be the same as in.NumFrames(), but other values are possible if some layers do splicing.

Implements Component.

Definition at line 3531 of file nnet-component.cc.

References CuMatrixBase< Real >::Add(), CuMatrixBase< Real >::ApplyHeaviside(), ChunkInfo::CheckSize(), DropoutComponent::dropout_proportion_, DropoutComponent::dropout_scale_, DropoutComponent::InputDim(), KALDI_ASSERT, CuMatrixBase< Real >::MulElements(), ChunkInfo::NumChunks(), CuMatrixBase< Real >::NumCols(), RandomComponent::random_generator_, kaldi::RandUniform(), and CuMatrixBase< Real >::Scale().

3534  {
3535  in_info.CheckSize(in);
3536  out_info.CheckSize(*out);
3537  KALDI_ASSERT(in_info.NumChunks() == out_info.NumChunks());
3538  KALDI_ASSERT(in.NumCols() == this->InputDim());
3539 
3541  KALDI_ASSERT(dp < 1.0 && dp >= 0.0);
3542  KALDI_ASSERT(dropout_scale_ <= 1.0 && dropout_scale_ >= 0.0);
3543 
3544  BaseFloat low_scale = dropout_scale_,
3545  high_scale = (1.0 - (dp * low_scale)) / (1.0 - dp),
3546  average = (low_scale * dp) +
3547  (high_scale * (1.0 - dp));
3548  KALDI_ASSERT(fabs(average - 1.0) < 0.01);
3549 
3550  // This const_cast is only safe assuming you don't attempt
3551  // to use multi-threaded code with the GPU.
3552  const_cast<CuRand<BaseFloat>&>(random_generator_).RandUniform(out);
3553 
3554 
3555  out->Add(-dp); // now, a proportion "dp" will be <0.0
3556  out->ApplyHeaviside(); // apply the function (x>0?1:0). Now, a proportion "dp" will
3557  // be zero and (1-dp) will be 1.0.
3558  if ((high_scale - low_scale) != 1.0)
3559  out->Scale(high_scale - low_scale); // now, "dp" are 0 and (1-dp) are "high_scale-low_scale".
3560  if (low_scale != 0.0)
3561  out->Add(low_scale); // now "dp" equal "low_scale" and (1.0-dp) equal "high_scale".
3562 
3563  out->MulElements(in);
3564 }
float RandUniform(struct RandomState *state=NULL)
Returns a random number strictly between 0 and 1.
Definition: kaldi-math.h:151
void Scale(Real value)
Definition: cu-matrix.cc:608
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
virtual int32 InputDim() const
Get size of input vectors.
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void MulElements(const CuMatrixBase< Real > &A)
Multiply two matrices elementwise: C = C .* A.
Definition: cu-matrix.cc:652
void ApplyHeaviside()
For each element, sets x = (x > 0 ? 1.0 : 0.0).
Definition: cu-matrix.cc:2309
void Add(Real value)
Definition: cu-matrix.cc:546
CuRand< BaseFloat > random_generator_
void Read ( std::istream &  is,
bool  binary 
)
virtual

Implements Component.

Definition at line 3501 of file nnet-component.cc.

References DropoutComponent::dim_, DropoutComponent::dropout_proportion_, DropoutComponent::dropout_scale_, kaldi::nnet2::ExpectOneOrTwoTokens(), kaldi::ExpectToken(), and kaldi::ReadBasicType().

3501  {
3502  ExpectOneOrTwoTokens(is, binary, "<DropoutComponent>", "<Dim>");
3503  ReadBasicType(is, binary, &dim_);
3504  ExpectToken(is, binary, "<DropoutScale>");
3505  ReadBasicType(is, binary, &dropout_scale_);
3506  ExpectToken(is, binary, "<DropoutProportion>");
3507  ReadBasicType(is, binary, &dropout_proportion_);
3508  ExpectToken(is, binary, "</DropoutComponent>");
3509 }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
static void ExpectOneOrTwoTokens(std::istream &is, bool binary, const std::string &token1, const std::string &token2)
void SetDropoutScale ( BaseFloat  scale)
inline

Definition at line 1609 of file nnet-component.h.

References DropoutComponent::dropout_scale_.

Referenced by Nnet::SetDropoutScale().

1609 { dropout_scale_ = scale; }
virtual std::string Type ( ) const
inlinevirtual

Implements Component.

Definition at line 1607 of file nnet-component.h.

1607 { return "DropoutComponent"; }
void Write ( std::ostream &  os,
bool  binary 
) const
virtual

Write component to stream.

Implements Component.

Definition at line 3511 of file nnet-component.cc.

References DropoutComponent::dim_, DropoutComponent::dropout_proportion_, DropoutComponent::dropout_scale_, kaldi::WriteBasicType(), and kaldi::WriteToken().

3511  {
3512  WriteToken(os, binary, "<DropoutComponent>");
3513  WriteToken(os, binary, "<Dim>");
3514  WriteBasicType(os, binary, dim_);
3515  WriteToken(os, binary, "<DropoutScale>");
3516  WriteBasicType(os, binary, dropout_scale_);
3517  WriteToken(os, binary, "<DropoutProportion>");
3518  WriteBasicType(os, binary, dropout_proportion_);
3519  WriteToken(os, binary, "</DropoutComponent>");
3520 }
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34

Member Data Documentation


The documentation for this class was generated from the following files: