#include <nnet-component.h>

Inheritance diagram for BlockAffineComponentPreconditioned:

Collaboration diagram for BlockAffineComponentPreconditioned:

[legend]

Public Member Functions
void	Init (BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 num_blocks, BaseFloat alpha)

virtual void	InitFromString (std::string args)
	Initialize, typically from a line of a config file. More...

	BlockAffineComponentPreconditioned ()

virtual std::string	Type () const

virtual void	SetZero (bool treat_as_gradient)
	Set parameters to zero, and if treat_as_gradient is true, we'll be treating this as a gradient so set the learning rate to 1 and make any other changes necessary (there's a variable we have to set for the MixtureProbComponent). More...

virtual void	Read (std::istream &is, bool binary)

virtual void	Write (std::ostream &os, bool binary) const
	Write component to stream. More...

virtual Component *	Copy () const
	Copy component (deep copy). More...

Public Member Functions inherited from BlockAffineComponent
virtual int32	InputDim () const
	Get size of input vectors. More...

virtual int32	OutputDim () const
	Get size of output vectors. More...

virtual int32	GetParameterDim () const
	The following new virtual function returns the total dimension of the parameters in this class. More...

virtual void	Vectorize (VectorBase< BaseFloat > *params) const
	Turns the parameters into vector form. More...

virtual void	UnVectorize (const VectorBase< BaseFloat > &params)
	Converts the parameters from vector form. More...

void	Init (BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 num_blocks)

	BlockAffineComponent ()

virtual bool	BackpropNeedsInput () const

virtual bool	BackpropNeedsOutput () const

virtual void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const
	Perform forward pass propagation Input->Output. More...

virtual void	Backprop (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component to_update, CuMatrix< BaseFloat > in_deriv) const
	Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise). More...

virtual BaseFloat	DotProduct (const UpdatableComponent &other) const
	Here, "other" is a component of the same specific type. More...

virtual void	PerturbParams (BaseFloat stddev)
	We introduce a new virtual function that only applies to class UpdatableComponent. More...

virtual void	Scale (BaseFloat scale)
	This new virtual function scales the parameters by this amount. More...

virtual void	Add (BaseFloat alpha, const UpdatableComponent &other)
	This new virtual function adds the parameters of another updatable component, times some constant, to the current parameters. More...

Public Member Functions inherited from UpdatableComponent
	UpdatableComponent (const UpdatableComponent &other)

void	Init (BaseFloat learning_rate)

	UpdatableComponent (BaseFloat learning_rate)

	UpdatableComponent ()

virtual	~UpdatableComponent ()

void	SetLearningRate (BaseFloat lrate)
	Sets the learning rate of gradient descent. More...

BaseFloat	LearningRate () const
	Gets the learning rate of gradient descent. More...

virtual std::string	Info () const

Public Member Functions inherited from Component
	Component ()

virtual int32	Index () const
	Returns the index in the sequence of layers in the neural net; intended only to be used in debugging information. More...

virtual void	SetIndex (int32 index)

virtual std::vector< int32 >	Context () const
	Return a vector describing the temporal context this component requires for each frame of output, as a sorted list. More...

void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out) const
	A non-virtual propagate function that first resizes output if necessary. More...

virtual	~Component ()

Private Member Functions
	KALDI_DISALLOW_COPY_AND_ASSIGN (BlockAffineComponentPreconditioned)

virtual void	Update (const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_deriv)

Private Attributes
bool	is_gradient_

BaseFloat	alpha_

Additional Inherited Members
Static Public Member Functions inherited from Component
static Component *	ReadNew (std::istream &is, bool binary)
	Read component from stream. More...

static Component *	NewFromString (const std::string &initializer_line)
	Initialize the Component from one line that will contain first the type, e.g. More...

static Component *	NewComponentOfType (const std::string &type)
	Return a new Component of the given type e.g. More...

Protected Member Functions inherited from BlockAffineComponent
virtual void	UpdateSimple (const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_deriv)

Protected Attributes inherited from BlockAffineComponent
CuMatrix< BaseFloat >	linear_params_

CuVector< BaseFloat >	bias_params_

int32	num_blocks_

Protected Attributes inherited from UpdatableComponent
BaseFloat	learning_rate_
	learning rate (0.0..0.01) More...

Detailed Description

Definition at line 1242 of file nnet-component.h.

Constructor & Destructor Documentation

◆ BlockAffineComponentPreconditioned()

BlockAffineComponentPreconditioned ( )

inline

Definition at line 1252 of file nnet-component.h.

1252 { } // use Init to really initialize.

Member Function Documentation

◆ Copy()

Component * Copy ( ) const

virtual

Copy component (deep copy).

Reimplemented from BlockAffineComponent.

Definition at line 2241 of file nnet-component.cc.

References AffineComponentPreconditionedOnline::alpha_, BlockAffineComponentPreconditioned::alpha_, AffineComponent::bias_params_, BlockAffineComponent::bias_params_, AffineComponent::is_gradient_, BlockAffineComponentPreconditioned::is_gradient_, UpdatableComponent::learning_rate_, AffineComponent::linear_params_, BlockAffineComponent::linear_params_, and BlockAffineComponent::num_blocks_.

                                                           {
   BlockAffineComponentPreconditioned *ans = new
       BlockAffineComponentPreconditioned();
   ans->learning_rate_ = learning_rate_;
   ans->linear_params_ = linear_params_;
   ans->bias_params_ = bias_params_;
   ans->num_blocks_ = num_blocks_;
   ans->alpha_ = alpha_;
   ans->is_gradient_ = is_gradient_;
   return ans;
 }

◆ Init()

void Init	(	BaseFloat	learning_rate,
		int32	input_dim,
		int32	output_dim,
		BaseFloat	param_stddev,
		BaseFloat	bias_stddev,
		int32	num_blocks,
		BaseFloat	alpha
	)

Definition at line 2161 of file nnet-component.cc.

References AffineComponentPreconditionedOnline::alpha_, BlockAffineComponent::Init(), AffineComponent::is_gradient_, and KALDI_ASSERT.

Referenced by kaldi::nnet2::UnitTestBlockAffineComponentPreconditioned().

                                                                {
   BlockAffineComponent::Init(learning_rate, input_dim, output_dim,
                              param_stddev, bias_stddev, num_blocks);
   is_gradient_ = false;
   KALDI_ASSERT(alpha > 0.0);
   alpha_ = alpha;
 }

◆ InitFromString()

void InitFromString ( std::string args )

virtual

Initialize, typically from a line of a config file.

The "args" will contain any parameters that need to be passed to the Component, e.g. dimensions.

Reimplemented from BlockAffineComponent.

Definition at line 2174 of file nnet-component.cc.

References AffineComponentPreconditionedOnline::Init(), KALDI_ERR, UpdatableComponent::learning_rate_, and kaldi::nnet2::ParseFromString().

Referenced by kaldi::nnet2::UnitTestBlockAffineComponentPreconditioned().

                                                                       {
   std::string orig_args(args);
   bool ok = true;
   BaseFloat learning_rate = learning_rate_;
   BaseFloat alpha = 4.0;
   int32 input_dim = -1, output_dim = -1, num_blocks = 1;
   ParseFromString("learning-rate", &args, &learning_rate); // optional.
   ParseFromString("alpha", &args, &alpha);
   ok = ok && ParseFromString("input-dim", &args, &input_dim);
   ok = ok && ParseFromString("output-dim", &args, &output_dim);
   ok = ok && ParseFromString("num-blocks", &args, &num_blocks);
 
   BaseFloat param_stddev = 1.0 / std::sqrt(input_dim),
       bias_stddev = 1.0;
   ParseFromString("param-stddev", &args, &param_stddev);
   ParseFromString("bias-stddev", &args, &bias_stddev);
   if (!args.empty())
     KALDI_ERR << "Could not process these elements in initializer: "
               << args;
   if (!ok)
     KALDI_ERR << "Bad initializer " << orig_args;
   Init(learning_rate, input_dim, output_dim,
        param_stddev, bias_stddev, num_blocks,
        alpha);
 }

◆ KALDI_DISALLOW_COPY_AND_ASSIGN()

KALDI_DISALLOW_COPY_AND_ASSIGN ( BlockAffineComponentPreconditioned )

private

◆ Read()

void Read	(	std::istream &	is,
		bool	binary
	)

virtual

Reimplemented from BlockAffineComponent.

Definition at line 2206 of file nnet-component.cc.

References AffineComponentPreconditionedOnline::alpha_, AffineComponent::bias_params_, kaldi::nnet2::ExpectOneOrTwoTokens(), kaldi::ExpectToken(), AffineComponent::is_gradient_, UpdatableComponent::learning_rate_, AffineComponent::linear_params_, and kaldi::ReadBasicType().

                                                                          {
   ExpectOneOrTwoTokens(is, binary, "<BlockAffineComponentPreconditioned>",
                        "<LearningRate>");
   ReadBasicType(is, binary, &learning_rate_);
   ExpectToken(is, binary, "<NumBlocks>");
   ReadBasicType(is, binary, &num_blocks_);
   ExpectToken(is, binary, "<LinearParams>");
   linear_params_.Read(is, binary);
   ExpectToken(is, binary, "<BiasParams>");
   bias_params_.Read(is, binary);
   ExpectToken(is, binary, "<Alpha>");
   ReadBasicType(is, binary, &alpha_);
   ExpectToken(is, binary, "<IsGradient>");
   ReadBasicType(is, binary, &is_gradient_);
   ExpectToken(is, binary, "</BlockAffineComponentPreconditioned>");
 }

◆ SetZero()

void SetZero ( bool treat_as_gradient )

virtual

Set parameters to zero, and if treat_as_gradient is true, we'll be treating this as a gradient so set the learning rate to 1 and make any other changes necessary (there's a variable we have to set for the MixtureProbComponent).

Reimplemented from BlockAffineComponent.

Definition at line 2200 of file nnet-component.cc.

References AffineComponent::is_gradient_, and BlockAffineComponent::SetZero().

                                                                        {
   if (treat_as_gradient)
     is_gradient_ = true;
   BlockAffineComponent::SetZero(treat_as_gradient);
 }

◆ Type()

virtual std::string Type ( ) const

inlinevirtual

Reimplemented from BlockAffineComponent.

Definition at line 1253 of file nnet-component.h.

References kaldi::cu::Copy(), and KALDI_DISALLOW_COPY_AND_ASSIGN.

1253 { return "BlockAffineComponentPreconditioned"; }

◆ Update()

void Update	(	const CuMatrixBase< BaseFloat > &	in_value,
		const CuMatrixBase< BaseFloat > &	out_deriv
	)

privatevirtual

Reimplemented from BlockAffineComponent.

Definition at line 2253 of file nnet-component.cc.

References CuMatrixBase< Real >::AddMatMat(), AffineComponentPreconditionedOnline::alpha_, AffineComponent::bias_params_, CuVectorBase< Real >::CopyColFromMat(), CuMatrixBase< Real >::CopyFromMat(), AffineComponent::is_gradient_, kaldi::kNoTrans, kaldi::kTrans, kaldi::kUndefined, UpdatableComponent::learning_rate_, AffineComponent::linear_params_, CuMatrixBase< Real >::NumRows(), kaldi::nnet2::PreconditionDirectionsAlphaRescaled(), and AffineComponent::UpdateSimple().

                                               {
   if (is_gradient_) {
     UpdateSimple(in_value, out_deriv);
     // does the baseline update with no preconditioning.
     return;
   }
   int32 input_block_dim = linear_params_.NumCols(),
       output_block_dim = linear_params_.NumRows() / num_blocks_,
       num_frames = in_value.NumRows();
 
   CuMatrix<BaseFloat> in_value_temp(num_frames, input_block_dim + 1, kUndefined),
       in_value_precon(num_frames, input_block_dim + 1, kUndefined);
   in_value_temp.Set(1.0); // so last row will have value 1.0.
   CuSubMatrix<BaseFloat> in_value_temp_part(in_value_temp, 0, num_frames,
                                             0, input_block_dim); // all but last 1.0
   CuSubMatrix<BaseFloat> in_value_precon_part(in_value_precon, 0, num_frames,
                                             0, input_block_dim);
   CuVector<BaseFloat> precon_ones(num_frames);
   CuMatrix<BaseFloat> out_deriv_precon(num_frames, output_block_dim, kUndefined);
 
   for (int32 b = 0; b < num_blocks_; b++) {
     CuSubMatrix<BaseFloat> in_value_block(in_value, 0, num_frames,
                                         b * input_block_dim,
                                         input_block_dim),
         out_deriv_block(out_deriv, 0, num_frames,
                         b * output_block_dim, output_block_dim),
         param_block(linear_params_,
                     b * output_block_dim, output_block_dim,
                     0, input_block_dim);
     in_value_temp_part.CopyFromMat(in_value_block);
 
     PreconditionDirectionsAlphaRescaled(in_value_temp, alpha_,
                                         &in_value_precon);
     PreconditionDirectionsAlphaRescaled(out_deriv_block, alpha_,
                                         &out_deriv_precon);
 
 
     // Update the parameters.
     param_block.AddMatMat(learning_rate_, out_deriv_precon, kTrans,
                           in_value_precon_part, kNoTrans, 1.0);
     precon_ones.CopyColFromMat(in_value_precon, input_block_dim);
     bias_params_.Range(b * output_block_dim, output_block_dim).
         AddMatVec(learning_rate_, out_deriv_precon, kTrans,
                   precon_ones, 1.0);
   }
 }

◆ Write()

void Write	(	std::ostream &	os,
		bool	binary
	)		const

virtual

Write component to stream.

Reimplemented from BlockAffineComponent.

Definition at line 2223 of file nnet-component.cc.

References AffineComponentPreconditionedOnline::alpha_, AffineComponent::bias_params_, AffineComponent::is_gradient_, UpdatableComponent::learning_rate_, AffineComponent::linear_params_, kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                                   {
   WriteToken(os, binary, "<BlockAffineComponentPreconditioned>");
   WriteToken(os, binary, "<LearningRate>");
   WriteBasicType(os, binary, learning_rate_);
   WriteToken(os, binary, "<NumBlocks>");
   WriteBasicType(os, binary, num_blocks_);
   WriteToken(os, binary, "<LinearParams>");
   linear_params_.Write(os, binary);
   WriteToken(os, binary, "<BiasParams>");
   bias_params_.Write(os, binary);
   WriteToken(os, binary, "<Alpha>");
   WriteBasicType(os, binary, alpha_);
   WriteToken(os, binary, "<IsGradient>");
   WriteBasicType(os, binary, is_gradient_);
   WriteToken(os, binary, "</BlockAffineComponentPreconditioned>");
 }

Member Data Documentation

◆ alpha_

BaseFloat alpha_

private

Definition at line 1265 of file nnet-component.h.

Referenced by BlockAffineComponentPreconditioned::Copy().

◆ is_gradient_

bool is_gradient_

private

Definition at line 1264 of file nnet-component.h.

Referenced by BlockAffineComponentPreconditioned::Copy().

The documentation for this class was generated from the following files:

nnet2/nnet-component.h
nnet2/nnet-component.cc

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ BlockAffineComponentPreconditioned()

Member Function Documentation

◆ Copy()

◆ Init()

◆ InitFromString()

◆ KALDI_DISALLOW_COPY_AND_ASSIGN()

◆ Read()

◆ SetZero()

◆ Type()

◆ Update()

◆ Write()

Member Data Documentation

◆ alpha_

◆ is_gradient_