#include <nnet-activation.h>

Inheritance diagram for Dropout:

Collaboration diagram for Dropout:

[legend]

Public Member Functions
	Dropout (int32 dim_in, int32 dim_out)

	~Dropout ()

Component *	Copy () const
	Copy component (deep copy),. More...

ComponentType	GetType () const
	Get Type Identification of the component,. More...

void	InitData (std::istream &is)
	Virtual interface for initialization and I/O,. More...

void	ReadData (std::istream &is, bool binary)
	Reads the component content. More...

void	WriteData (std::ostream &os, bool binary) const
	Writes the component content. More...

std::string	Info () const
	Print some additional info (after <ComponentName> and the dims),. More...

void	PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
	Abstract interface for propagation/backpropagation. More...

void	BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
	Backward pass transformation (to be implemented by descending class...) More...

BaseFloat	GetDropoutRate ()

void	SetDropoutRate (BaseFloat dr)

Public Member Functions inherited from Component
	Component (int32 input_dim, int32 output_dim)
	Generic interface of a component,. More...

virtual	~Component ()

virtual bool	IsUpdatable () const
	Check if componeny has 'Updatable' interface (trainable components),. More...

virtual bool	IsMultistream () const
	Check if component has 'Recurrent' interface (trainable and recurrent),. More...

int32	InputDim () const
	Get the dimension of the input,. More...

int32	OutputDim () const
	Get the dimension of the output,. More...

void	Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward-pass propagation 'in' -> 'out',. More...

void	Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
	Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...

void	Write (std::ostream &os, bool binary) const
	Write the component to a stream,. More...

virtual std::string	InfoGradient () const
	Print some additional info about gradient (after <...> and dims),. More...

Private Attributes
BaseFloat	dropout_rate_
	probability that a neuron is dropped, More...

CuRand< BaseFloat >	rand_
	generator of random numbers, More...

CuMatrix< BaseFloat >	dropout_mask_

Additional Inherited Members
Public Types inherited from Component
enum	ComponentType { kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform, kConvolutionalComponent, kLstmProjected, kBlstmProjected, kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax, kBlockSoftmax, kSigmoid, kTanh, kParametricRelu, kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm, kSplice, kCopy, kTranspose, kBlockLinearity, kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent, kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kMaxPoolingComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent }
	Component type identification mechanism,. More...

Static Public Member Functions inherited from Component
static const char *	TypeToMarker (ComponentType t)
	Converts component type to marker,. More...

static ComponentType	MarkerToType (const std::string &s)
	Converts marker to component type (case insensitive),. More...

static Component *	Init (const std::string &conf_line)
	Initialize component from a line in config file,. More...

static Component *	Read (std::istream &is, bool binary)
	Read the component from a stream (static method),. More...

Static Public Attributes inherited from Component
static const struct key_value	kMarkerMap []
	The table with pairs of Component types and markers (defined in nnet-component.cc),. More...

Protected Attributes inherited from Component
int32	input_dim_
	Data members,. More...

int32	output_dim_
	Dimension of the output of the Component,. More...

Detailed Description

Definition at line 269 of file nnet-activation.h.

Constructor & Destructor Documentation

◆ Dropout()

Dropout	(	int32	dim_in,
		int32	dim_out
	)

inline

Definition at line 271 of file nnet-activation.h.

                                       :
       Component(dim_in, dim_out),
       dropout_rate_(0.5)
   { }

◆ ~Dropout()

~Dropout ( )

inline

Definition at line 276 of file nnet-activation.h.

277 { }

Member Function Documentation

◆ BackpropagateFnc()

void BackpropagateFnc	(	const CuMatrixBase< BaseFloat > &	in,
		const CuMatrixBase< BaseFloat > &	out,
		const CuMatrixBase< BaseFloat > &	out_diff,
		CuMatrixBase< BaseFloat > *	in_diff
	)

inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 342 of file nnet-activation.h.

References CuMatrixBase< Real >::CopyFromMat(), CuMatrixBase< Real >::MulElements(), and CuMatrixBase< Real >::Scale().

                                                           {
     in_diff->CopyFromMat(out_diff);
     // use same mask on the error derivatives...
     in_diff->MulElements(dropout_mask_);
     // enlarge the output to fit same dynamic range as w/o dropout
     in_diff->Scale(1.0 / (1.0 - dropout_rate_));
   }

◆ Copy()

Component* Copy ( ) const

inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 279 of file nnet-activation.h.

279 { return new Dropout(*this); }

kaldi::nnet1::Dropout::Dropout

Dropout(int32 dim_in, int32 dim_out)

Definition: nnet-activation.h:271

◆ GetDropoutRate()

BaseFloat GetDropoutRate ( )

inline

Definition at line 353 of file nnet-activation.h.

Referenced by Nnet::SetDropoutRate().

353 { return dropout_rate_; }

kaldi::nnet1::Dropout::dropout_rate_

BaseFloat dropout_rate_

probability that a neuron is dropped,

Definition: nnet-activation.h:361

◆ GetType()

ComponentType GetType ( ) const

inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 280 of file nnet-activation.h.

References Component::kDropout.

280 { return kDropout; }

kaldi::nnet1::Component::kDropout

Definition: nnet-component.h:65

◆ Info()

std::string Info ( ) const

inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 325 of file nnet-activation.h.

References kaldi::nnet1::ToString().

                          {
     return std::string("<DropoutRate> ") + ToString(dropout_rate_);
   }

◆ InitData()

void InitData ( std::istream & is )

inlinevirtual

Virtual interface for initialization and I/O,.

Initialize internal data of a component

Reimplemented from Component.

Definition at line 282 of file nnet-activation.h.

References KALDI_ASSERT, KALDI_ERR, kaldi::ReadBasicType(), and kaldi::ReadToken().

                                 {
     is >> std::ws;  // eat-up whitespace
     // parse config
     std::string token;
     while (is >> std::ws, !is.eof()) {
       ReadToken(is, false, &token);
        if (token == "<DropoutRate>") ReadBasicType(is, false, &dropout_rate_);
       else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
                      << " (DropoutRate)";
     }
     KALDI_ASSERT(dropout_rate_ >= 0.0 && dropout_rate_ < 1.0);
   }

◆ PropagateFnc()

void PropagateFnc	(	const CuMatrixBase< BaseFloat > &	in,
		CuMatrixBase< BaseFloat > *	out
	)

inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 329 of file nnet-activation.h.

References CuMatrixBase< Real >::CopyFromMat(), CuMatrixBase< Real >::MulElements(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), and CuMatrixBase< Real >::Scale().

                                                   {
     out->CopyFromMat(in);
     // set N inputs to zero, according to the 'dropout_rate_' ...
     dropout_mask_.Resize(out->NumRows(), out->NumCols());
     rand_.RandUniform(&dropout_mask_);  // [0..1]
     dropout_mask_.Add(-dropout_rate_);  // [(-rate)..(1-rate)]
     dropout_mask_.Heaviside(dropout_mask_); // (x > 0.0 ? 1 : 0)
     out->MulElements(dropout_mask_);
     // rescale to keep the same dynamic range as w/o dropout,
     out->Scale(1.0 / (1.0 - dropout_rate_));
   }

◆ ReadData()

void ReadData	(	std::istream &	is,
		bool	binary
	)

inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 295 of file nnet-activation.h.

References kaldi::ExpectToken(), KALDI_ASSERT, KALDI_ERR, kaldi::Peek(), kaldi::PeekToken(), kaldi::ReadBasicType(), and kaldi::ReadToken().

                                              {
     // Read all the '<Tokens>' in arbitrary order,
     bool finished = false;
     while ('<' == Peek(is, binary) && !finished) {
       std::string token;
       int first_char = PeekToken(is, binary);
       switch (first_char) {
         case 'D': ReadToken(is, false, &token);
            if (token == "<DropoutRate>") ReadBasicType(is, binary, &dropout_rate_);
           else if (token == "<DropoutRetention>") { /* compatibility */
             BaseFloat dropout_retention;
             ReadBasicType(is, binary, &dropout_retention);
             dropout_rate_ = 1.0 - dropout_retention;
           } else KALDI_ERR << "Unknown token: " << token;
           break;
         case '!': ExpectToken(is, binary, "<!EndOfComponent>");
           finished = true;
           break;
         default: ReadToken(is, false, &token);
           KALDI_ERR << "Unknown token: " << token;
       }
     }
     KALDI_ASSERT(dropout_rate_ >= 0.0 && dropout_rate_ < 1.0);
   }

◆ SetDropoutRate()

void SetDropoutRate ( BaseFloat dr )

inline

Definition at line 355 of file nnet-activation.h.

References KALDI_ASSERT.

Referenced by Nnet::SetDropoutRate().

                                     {
     dropout_rate_ = dr;
     KALDI_ASSERT(dropout_rate_ >= 0.0 && dropout_rate_ < 1.0);
   }

◆ WriteData()

void WriteData	(	std::ostream &	os,
		bool	binary
	)		const

inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 320 of file nnet-activation.h.

References kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                     {
     WriteToken(os, binary, "<DropoutRate>");
     WriteBasicType(os, binary, dropout_rate_);
   }

Member Data Documentation

◆ dropout_mask_

CuMatrix<BaseFloat> dropout_mask_

private

Definition at line 365 of file nnet-activation.h.

◆ dropout_rate_

BaseFloat dropout_rate_

private

probability that a neuron is dropped,

Definition at line 361 of file nnet-activation.h.

◆ rand_

CuRand<BaseFloat> rand_

private

generator of random numbers,

Definition at line 363 of file nnet-activation.h.

The documentation for this class was generated from the following file:

nnet/nnet-activation.h

Public Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ Dropout()

◆ ~Dropout()

Member Function Documentation

◆ BackpropagateFnc()

◆ Copy()

◆ GetDropoutRate()

◆ GetType()

◆ Info()

◆ InitData()

◆ PropagateFnc()

◆ ReadData()

◆ SetDropoutRate()

◆ WriteData()

Member Data Documentation

◆ dropout_mask_

◆ dropout_rate_

◆ rand_