SimpleSentenceAveragingComponent does not have nested network, it is intended to be used inside of a <ParallelComponent>. More...

#include <nnet-sentence-averaging-component.h>

Inheritance diagram for SimpleSentenceAveragingComponent:

[legend]

Collaboration diagram for SimpleSentenceAveragingComponent:

[legend]

Public Member Functions
	SimpleSentenceAveragingComponent (int32 dim_in, int32 dim_out)

	~SimpleSentenceAveragingComponent ()

Component *	Copy () const
	Copy component (deep copy),. More...

ComponentType	GetType () const
	Get Type Identification of the component,. More...

void	InitData (std::istream &is)
	Virtual interface for initialization and I/O,. More...

void	ReadData (std::istream &is, bool binary)
	Reads the component content. More...

void	WriteData (std::ostream &os, bool binary) const
	Writes the component content. More...

std::string	Info () const
	Print some additional info (after <ComponentName> and the dims),. More...

std::string	InfoGradient () const
	Print some additional info about gradient (after <...> and dims),. More...

void	PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
	Abstract interface for propagation/backpropagation. More...

void	BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
	Backward pass transformation (to be implemented by descending class...) More...

Public Member Functions inherited from Component
	Component (int32 input_dim, int32 output_dim)
	Generic interface of a component,. More...

virtual	~Component ()

virtual bool	IsUpdatable () const
	Check if componeny has 'Updatable' interface (trainable components),. More...

virtual bool	IsMultistream () const
	Check if component has 'Recurrent' interface (trainable and recurrent),. More...

int32	InputDim () const
	Get the dimension of the input,. More...

int32	OutputDim () const
	Get the dimension of the output,. More...

void	Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward-pass propagation 'in' -> 'out',. More...

void	Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
	Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...

void	Write (std::ostream &os, bool binary) const
	Write the component to a stream,. More...

Private Attributes
CuVector< BaseFloat >	average_row_
	Auxiliary buffer for forward propagation (for average vector),. More...

CuVector< BaseFloat >	average_diff_
	Auxiliary buffer for back-propagation (for average vector),. More...

BaseFloat	gradient_boost_
	Scalar applied on gradient in backpropagation,. More...

BaseFloat	shrinkage_
	Number of 'imaginary' zero-vectors in the average (shrinks the average vector for short sentences),. More...

bool	only_summing_
	Removes normalization term from arithmetic mean (when true). More...

Additional Inherited Members
Public Types inherited from Component
enum	ComponentType { kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform, kConvolutionalComponent, kLstmProjected, kBlstmProjected, kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax, kBlockSoftmax, kSigmoid, kTanh, kParametricRelu, kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm, kSplice, kCopy, kTranspose, kBlockLinearity, kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent, kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kMaxPoolingComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent }
	Component type identification mechanism,. More...

Static Public Member Functions inherited from Component
static const char *	TypeToMarker (ComponentType t)
	Converts component type to marker,. More...

static ComponentType	MarkerToType (const std::string &s)
	Converts marker to component type (case insensitive),. More...

static Component *	Init (const std::string &conf_line)
	Initialize component from a line in config file,. More...

static Component *	Read (std::istream &is, bool binary)
	Read the component from a stream (static method),. More...

Static Public Attributes inherited from Component
static const struct key_value	kMarkerMap []
	The table with pairs of Component types and markers (defined in nnet-component.cc),. More...

Protected Attributes inherited from Component
int32	input_dim_
	Data members,. More...

int32	output_dim_
	Dimension of the output of the Component,. More...

Detailed Description

SimpleSentenceAveragingComponent does not have nested network, it is intended to be used inside of a <ParallelComponent>.

For training use 'nnet-train-perutt'.

The sentence-averaging typically leads to small gradients, so we boost it 100x by default (boost = multiply, it's equivalent to applying learning-rate factor).

Definition at line 42 of file nnet-sentence-averaging-component.h.

Constructor & Destructor Documentation

◆ SimpleSentenceAveragingComponent()

SimpleSentenceAveragingComponent	(	int32	dim_in,
		int32	dim_out
	)

inline

Definition at line 44 of file nnet-sentence-averaging-component.h.

Referenced by SimpleSentenceAveragingComponent::Copy().

                                                                :
     Component(dim_in, dim_out),
     gradient_boost_(100.0),
     shrinkage_(0.0),
     only_summing_(false)
   { }

◆ ~SimpleSentenceAveragingComponent()

~SimpleSentenceAveragingComponent ( )

inline

Definition at line 51 of file nnet-sentence-averaging-component.h.

52 { }

Member Function Documentation

◆ BackpropagateFnc()

void BackpropagateFnc	(	const CuMatrixBase< BaseFloat > &	in,
		const CuMatrixBase< BaseFloat > &	out,
		const CuMatrixBase< BaseFloat > &	out_diff,
		CuMatrixBase< BaseFloat > *	in_diff
	)

inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 137 of file nnet-sentence-averaging-component.h.

References CuMatrixBase< Real >::AddVecToRows(), SimpleSentenceAveragingComponent::average_diff_, SimpleSentenceAveragingComponent::gradient_boost_, CuMatrixBase< Real >::NumRows(), SimpleSentenceAveragingComponent::only_summing_, Component::OutputDim(), and SimpleSentenceAveragingComponent::shrinkage_.

                                                           {
     // When averaging, a single frame from input influenced all the frames
     // on the output. So the derivative w.r.t. single input frame is a sum
     // of the output derivatives, scaled by the averaging constant 1/K.
     //
     // In the same time all the input frames of the average influenced
     // all the output frames. So the loss derivarive is same for all
     // the input frames coming to the averaging.
     //
     // getting the average output diff,
     average_diff_.Resize(OutputDim());
     if (only_summing_) {
       average_diff_.AddRowSumMat(1.0, out_diff, 0.0);
     } else {
       average_diff_.AddRowSumMat(1.0/(out_diff.NumRows()+shrinkage_), out_diff, 0.0);
     }
     // copy the derivative into the input diff, (applying gradient-boost!!)
     in_diff->AddVecToRows(gradient_boost_, average_diff_, 0.0);
   }

◆ Copy()

Component* Copy ( ) const

inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 54 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::SimpleSentenceAveragingComponent().

                           {
     return new SimpleSentenceAveragingComponent(*this);
   }

◆ GetType()

ComponentType GetType ( ) const

inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 58 of file nnet-sentence-averaging-component.h.

References Component::kSimpleSentenceAveragingComponent.

                                 {
     return kSimpleSentenceAveragingComponent;
   }

◆ Info()

std::string Info ( ) const

inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 115 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::gradient_boost_, SimpleSentenceAveragingComponent::only_summing_, SimpleSentenceAveragingComponent::shrinkage_, and kaldi::nnet1::ToString().

Referenced by SimpleSentenceAveragingComponent::InfoGradient().

                          {
     return std::string("\n  gradient-boost ") + ToString(gradient_boost_) +
       ", shrinkage: " + ToString(shrinkage_) +
       ", only summing: " + ToString(only_summing_);
   }

◆ InfoGradient()

std::string InfoGradient ( ) const

inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 120 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::Info().

                                  {
     return Info();
   }

◆ InitData()

void InitData ( std::istream & is )

inlinevirtual

Virtual interface for initialization and I/O,.

Initialize internal data of a component

Reimplemented from Component.

Definition at line 62 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::gradient_boost_, KALDI_ERR, SimpleSentenceAveragingComponent::only_summing_, kaldi::ReadBasicType(), kaldi::ReadToken(), and SimpleSentenceAveragingComponent::shrinkage_.

                                 {
     // parse config
     std::string token;
     while (is >> std::ws, !is.eof()) {
       ReadToken(is, false, &token);
       if (token == "<GradientBoost>") ReadBasicType(is, false, &gradient_boost_);
       else if (token == "<Shrinkage>") ReadBasicType(is, false, &shrinkage_);
       else if (token == "<OnlySumming>") ReadBasicType(is, false, &only_summing_);
       else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
                      << " (GradientBoost|Shrinkage|OnlySumming)";
     }
   }

◆ PropagateFnc()

void PropagateFnc	(	const CuMatrixBase< BaseFloat > &	in,
		CuMatrixBase< BaseFloat > *	out
	)

inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 124 of file nnet-sentence-averaging-component.h.

References CuMatrixBase< Real >::AddVecToRows(), SimpleSentenceAveragingComponent::average_row_, Component::InputDim(), CuMatrixBase< Real >::NumRows(), SimpleSentenceAveragingComponent::only_summing_, and SimpleSentenceAveragingComponent::shrinkage_.

                                                   {
     // get the average row-vector,
     average_row_.Resize(InputDim());
     if (only_summing_) {
       average_row_.AddRowSumMat(1.0, in, 0.0);
     } else {
       average_row_.AddRowSumMat(1.0/(in.NumRows()+shrinkage_), in, 0.0);
     }
     // copy it on the output,
     out->AddVecToRows(1.0, average_row_, 0.0);
   }

◆ ReadData()

void ReadData	(	std::istream &	is,
		bool	binary
	)

inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 75 of file nnet-sentence-averaging-component.h.

References kaldi::ExpectToken(), SimpleSentenceAveragingComponent::gradient_boost_, KALDI_WARN, SimpleSentenceAveragingComponent::only_summing_, kaldi::Peek(), kaldi::PeekToken(), kaldi::ReadBasicType(), and SimpleSentenceAveragingComponent::shrinkage_.

                                              {
     bool end_loop = false;
     while (!end_loop && '<' == Peek(is, binary)) {
       int first_char = PeekToken(is, binary);
       switch (first_char) {
         case 'G': ExpectToken(is, binary, "<GradientBoost>");
           ReadBasicType(is, binary, &gradient_boost_);
           break;
         case 'S': ExpectToken(is, binary, "<Shrinkage>");
           ReadBasicType(is, binary, &shrinkage_);
           break;
         case 'O': ExpectToken(is, binary, "<OnlySumming>");
           // compatibility trick,
           // in some models 'only_summing_' was float '0.0',
           // from now 'only_summing_' is 'bool':
           try {
             ReadBasicType(is, binary, &only_summing_);
           } catch(const std::exception &e) {
             KALDI_WARN << "ERROR was handled by exception!";
             BaseFloat dummy_float;
             ReadBasicType(is, binary, &dummy_float);
           }
           break;
         case '!':
           ExpectToken(is, binary, "<!EndOfComponent>");
         default:
           end_loop = true;
       }
     }
   }

◆ WriteData()

void WriteData	(	std::ostream &	os,
		bool	binary
	)		const

inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 106 of file nnet-sentence-averaging-component.h.

References SimpleSentenceAveragingComponent::gradient_boost_, SimpleSentenceAveragingComponent::only_summing_, SimpleSentenceAveragingComponent::shrinkage_, kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                     {
     WriteToken(os, binary, "<GradientBoost>");
     WriteBasicType(os, binary, gradient_boost_);
     WriteToken(os, binary, "<Shrinkage>");
     WriteBasicType(os, binary, shrinkage_);
     WriteToken(os, binary, "<OnlySumming>");
     WriteBasicType(os, binary, only_summing_);
   }