This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two. More...

#include <nnet-component.h>

Inheritance diagram for DropoutComponent:

Collaboration diagram for DropoutComponent:

[legend]

Public Member Functions
void	Init (int32 dim, BaseFloat dropout_proportion=0.5, BaseFloat dropout_scale=0.0)
	dropout-proportion is the proportion that is dropped out, e.g. More...

	DropoutComponent (int32 dim, BaseFloat dp=0.5, BaseFloat sc=0.0)

	DropoutComponent ()

virtual int32	InputDim () const
	Get size of input vectors. More...

virtual int32	OutputDim () const
	Get size of output vectors. More...

virtual void	InitFromString (std::string args)
	Initialize, typically from a line of a config file. More...

virtual void	Read (std::istream &is, bool binary)

virtual void	Write (std::ostream &os, bool binary) const
	Write component to stream. More...

virtual std::string	Type () const

void	SetDropoutScale (BaseFloat scale)

virtual bool	BackpropNeedsInput () const

virtual bool	BackpropNeedsOutput () const

virtual Component *	Copy () const
	Copy component (deep copy). More...

virtual void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const
	Perform forward pass propagation Input->Output. More...

virtual void	Backprop (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component to_update, CuMatrix< BaseFloat > in_deriv) const
	Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise). More...

virtual std::string	Info () const

Public Member Functions inherited from RandomComponent
void	ResetGenerator ()

Public Member Functions inherited from Component
	Component ()

virtual int32	Index () const
	Returns the index in the sequence of layers in the neural net; intended only to be used in debugging information. More...

virtual void	SetIndex (int32 index)

virtual std::vector< int32 >	Context () const
	Return a vector describing the temporal context this component requires for each frame of output, as a sorted list. More...

void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out) const
	A non-virtual propagate function that first resizes output if necessary. More...

virtual	~Component ()

Private Attributes
int32	dim_

BaseFloat	dropout_proportion_

BaseFloat	dropout_scale_

Additional Inherited Members
Static Public Member Functions inherited from Component
static Component *	ReadNew (std::istream &is, bool binary)
	Read component from stream. More...

static Component *	NewFromString (const std::string &initializer_line)
	Initialize the Component from one line that will contain first the type, e.g. More...

static Component *	NewComponentOfType (const std::string &type)
	Return a new Component of the given type e.g. More...

Protected Attributes inherited from RandomComponent
CuRand< BaseFloat >	random_generator_

Detailed Description

This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two.

Typically you would use this in training but not in test or when computing validation-set objective functions.

Definition at line 1584 of file nnet-component.h.

Constructor & Destructor Documentation

◆ DropoutComponent() [1/2]

DropoutComponent	(	int32	dim,
		BaseFloat	dp = `0.5`,
		BaseFloat	sc = `0.0`
	)

inline

Definition at line 1595 of file nnet-component.h.

                                                                       {
     Init(dim, dp, sc);
   }

◆ DropoutComponent() [2/2]

DropoutComponent ( )

inline

Definition at line 1598 of file nnet-component.h.

1598 : dim_(0), dropout_proportion_(0.5) { }

kaldi::nnet2::DropoutComponent::dropout_proportion_

BaseFloat dropout_proportion_

Definition: nnet-component.h:1628

kaldi::nnet2::DropoutComponent::dim_

int32 dim_

Definition: nnet-component.h:1627

Member Function Documentation

◆ Backprop()

void Backprop	(	const ChunkInfo &	in_info,
		const ChunkInfo &	out_info,
		const CuMatrixBase< BaseFloat > &	in_value,
		const CuMatrixBase< BaseFloat > &	out_value,
		const CuMatrixBase< BaseFloat > &	out_deriv,
		Component *	to_update,
		CuMatrix< BaseFloat > *	in_deriv
	)		const

virtual

Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise).

Note: in_value and out_value are the values of the input and output of the component, and these may be dummy variables if respectively BackpropNeedsInput() or BackpropNeedsOutput() return false for that component (not all components need these).

num_chunks lets us treat the input matrix as contiguous-in-time chunks of equal size; it only matters if splicing is involved.

Implements Component.

Definition at line 3566 of file nnet-component.cc.

References KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), CuMatrix< Real >::Resize(), kaldi::SameDim(), and CuMatrixBase< Real >::SetMatMatDivMat().

                                                                       {
   KALDI_ASSERT(SameDim(in_value, out_value) && SameDim(in_value, out_deriv));
   in_deriv->Resize(out_deriv.NumRows(), out_deriv.NumCols());
   in_deriv->SetMatMatDivMat(out_deriv, out_value, in_value);
 }

◆ BackpropNeedsInput()

virtual bool BackpropNeedsInput ( ) const

inlinevirtual

Reimplemented from Component.

Definition at line 1610 of file nnet-component.h.

1610 { return true; }

◆ BackpropNeedsOutput()

virtual bool BackpropNeedsOutput ( ) const

inlinevirtual

Reimplemented from Component.

Definition at line 1611 of file nnet-component.h.

References kaldi::cu::Copy(), and Component::Propagate().

1611 { return true; }

◆ Copy()

Component * Copy ( ) const

virtual

Copy component (deep copy).

Implements Component.

Definition at line 3578 of file nnet-component.cc.

                                         {
   return new DropoutComponent(dim_,
                               dropout_proportion_,
                               dropout_scale_);
 }

◆ Info()

std::string Info ( ) const

virtual

Reimplemented from Component.

Definition at line 3479 of file nnet-component.cc.

References Component::Info().

                                        {
   std::stringstream stream;
   stream << Component::Info() << ", dropout_proportion = "
          << dropout_proportion_ << ", dropout_scale = "
          << dropout_scale_;
   return stream.str();
 }

◆ Init()

void Init	(	int32	dim,
		BaseFloat	dropout_proportion = `0.5`,
		BaseFloat	dropout_scale = `0.0`
	)

dropout-proportion is the proportion that is dropped out, e.g.

if 0.1, we set 10% to a low value. [note, in some older code it was interpreted as the value not dropped out, so be careful.] The low scale-value is equal to dropout_scale. The high scale-value is chosen such that the expected scale-value is one.

Definition at line 3523 of file nnet-component.cc.

                                                     {
   dim_ = dim;
   dropout_proportion_ = dropout_proportion;
   dropout_scale_ = dropout_scale;
 }

◆ InitFromString()

void InitFromString ( std::string args )

virtual

Initialize, typically from a line of a config file.

The "args" will contain any parameters that need to be passed to the Component, e.g. dimensions.

Implements Component.

Definition at line 3487 of file nnet-component.cc.

References AffineComponentPreconditionedOnline::Init(), KALDI_ERR, and kaldi::nnet2::ParseFromString().

Referenced by kaldi::nnet2::UnitTestDropoutComponent().

                                                     {
   std::string orig_args(args);
   int32 dim;
   BaseFloat dropout_proportion = 0.5, dropout_scale = 0.0;
   bool ok = ParseFromString("dim", &args, &dim);
   ParseFromString("dropout-proportion", &args, &dropout_proportion);
   ParseFromString("dropout-scale", &args, &dropout_scale);
 
   if (!ok || !args.empty() || dim <= 0)
     KALDI_ERR << "Invalid initializer for layer of type DropoutComponent: \""
               << orig_args << "\"";
   Init(dim, dropout_proportion, dropout_scale);
 }

◆ InputDim()

virtual int32 InputDim ( ) const

inlinevirtual

Get size of input vectors.

Implements Component.

Definition at line 1599 of file nnet-component.h.

1599 { return dim_; }

kaldi::nnet2::DropoutComponent::dim_

int32 dim_

Definition: nnet-component.h:1627

◆ OutputDim()

virtual int32 OutputDim ( ) const

inlinevirtual

Get size of output vectors.

Implements Component.

Definition at line 1600 of file nnet-component.h.

1600 { return dim_; }

kaldi::nnet2::DropoutComponent::dim_

int32 dim_

Definition: nnet-component.h:1627

◆ Propagate()

void Propagate	(	const ChunkInfo &	in_info,
		const ChunkInfo &	out_info,
		const CuMatrixBase< BaseFloat > &	in,
		CuMatrixBase< BaseFloat > *	out
	)		const

virtual

Perform forward pass propagation Input->Output.

Each row is one frame or training example. Interpreted as "num_chunks" equally sized chunks of frames; this only matters for layers that do things like context splicing. Typically this variable will either be 1 (when we're processing a single contiguous chunk of data) or will be the same as in.NumFrames(), but other values are possible if some layers do splicing.

Implements Component.

Definition at line 3531 of file nnet-component.cc.

References CuMatrixBase< Real >::Add(), CuMatrixBase< Real >::ApplyHeaviside(), ChunkInfo::CheckSize(), AffineComponent::InputDim(), KALDI_ASSERT, CuMatrixBase< Real >::MulElements(), ChunkInfo::NumChunks(), CuMatrixBase< Real >::NumCols(), kaldi::RandUniform(), and CuMatrixBase< Real >::Scale().

                                                                       {
   in_info.CheckSize(in);
   out_info.CheckSize(*out);
   KALDI_ASSERT(in_info.NumChunks() == out_info.NumChunks());
   KALDI_ASSERT(in.NumCols() == this->InputDim());
 
   BaseFloat dp = dropout_proportion_;
   KALDI_ASSERT(dp < 1.0 && dp >= 0.0);
   KALDI_ASSERT(dropout_scale_ <= 1.0 && dropout_scale_ >= 0.0);
 
   BaseFloat low_scale = dropout_scale_,
       high_scale = (1.0 - (dp * low_scale)) / (1.0 - dp),
       average = (low_scale * dp) +
                 (high_scale * (1.0 - dp));
   KALDI_ASSERT(fabs(average - 1.0) < 0.01);
 
   // This const_cast is only safe assuming you don't attempt
   // to use multi-threaded code with the GPU.
   const_cast<CuRand<BaseFloat>&>(random_generator_).RandUniform(out);
 
 
   out->Add(-dp); // now, a proportion "dp" will be <0.0
   out->ApplyHeaviside(); // apply the function (x>0?1:0).  Now, a proportion "dp" will
                          // be zero and (1-dp) will be 1.0.
   if ((high_scale - low_scale) != 1.0)
     out->Scale(high_scale - low_scale); // now, "dp" are 0 and (1-dp) are "high_scale-low_scale".
   if (low_scale != 0.0)
     out->Add(low_scale); // now "dp" equal "low_scale" and (1.0-dp) equal "high_scale".
 
   out->MulElements(in);
 }

◆ Read()

void Read	(	std::istream &	is,
		bool	binary
	)

virtual

Implements Component.

Definition at line 3501 of file nnet-component.cc.

References kaldi::nnet2::ExpectOneOrTwoTokens(), kaldi::ExpectToken(), and kaldi::ReadBasicType().

                                                        {
   ExpectOneOrTwoTokens(is, binary, "<DropoutComponent>", "<Dim>");
   ReadBasicType(is, binary, &dim_);
   ExpectToken(is, binary, "<DropoutScale>");
   ReadBasicType(is, binary, &dropout_scale_);
   ExpectToken(is, binary, "<DropoutProportion>");
   ReadBasicType(is, binary, &dropout_proportion_);
   ExpectToken(is, binary, "</DropoutComponent>");
 }

◆ SetDropoutScale()

void SetDropoutScale ( BaseFloat scale )

inline

Definition at line 1609 of file nnet-component.h.

Referenced by Nnet::SetDropoutScale().

1609 { dropout_scale_ = scale; }

kaldi::nnet2::DropoutComponent::dropout_scale_

BaseFloat dropout_scale_

Definition: nnet-component.h:1629

◆ Type()

virtual std::string Type ( ) const

inlinevirtual

Implements Component.

Definition at line 1607 of file nnet-component.h.

1607 { return "DropoutComponent"; }

◆ Write()

void Write	(	std::ostream &	os,
		bool	binary
	)		const

virtual

Write component to stream.

Implements Component.

Definition at line 3511 of file nnet-component.cc.

References kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                               {
   WriteToken(os, binary, "<DropoutComponent>");
   WriteToken(os, binary, "<Dim>");
   WriteBasicType(os, binary, dim_);
   WriteToken(os, binary, "<DropoutScale>");
   WriteBasicType(os, binary, dropout_scale_);
   WriteToken(os, binary, "<DropoutProportion>");
   WriteBasicType(os, binary, dropout_proportion_);
   WriteToken(os, binary, "</DropoutComponent>");
 }

Member Data Documentation

◆ dim_

int32 dim_

private

Definition at line 1627 of file nnet-component.h.

◆ dropout_proportion_

BaseFloat dropout_proportion_

private

Definition at line 1628 of file nnet-component.h.

◆ dropout_scale_

BaseFloat dropout_scale_

private

Definition at line 1629 of file nnet-component.h.

The documentation for this class was generated from the following files:

nnet2/nnet-component.h
nnet2/nnet-component.cc

Public Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ DropoutComponent() [1/2]

◆ DropoutComponent() [2/2]

Member Function Documentation

◆ Backprop()

◆ BackpropNeedsInput()

◆ BackpropNeedsOutput()

◆ Copy()

◆ Info()

◆ Init()

◆ InitFromString()

◆ InputDim()

◆ OutputDim()

◆ Propagate()

◆ Read()

◆ SetDropoutScale()

◆ Type()

◆ Write()

Member Data Documentation

◆ dim_

◆ dropout_proportion_

◆ dropout_scale_