This kind of Component is a base-class for things like sigmoid and softmax. More...

#include <nnet-component.h>

Inheritance diagram for NonlinearComponent:

[legend]

Collaboration diagram for NonlinearComponent:

Public Member Functions
void	Init (int32 dim)

	NonlinearComponent (int32 dim)

	NonlinearComponent ()

	NonlinearComponent (const NonlinearComponent &other)

virtual int32	InputDim () const
	Get size of input vectors. More...

virtual int32	OutputDim () const
	Get size of output vectors. More...

virtual void	InitFromString (std::string args)
	We implement InitFromString at this level. More...

virtual void	Read (std::istream &is, bool binary)
	We implement Read at this level as it just needs the Type(). More...

virtual void	Write (std::ostream &os, bool binary) const
	Write component to stream. More...

void	Scale (BaseFloat scale)

void	Add (BaseFloat alpha, const NonlinearComponent &other)

const CuVector< double > &	ValueSum () const

const CuVector< double > &	DerivSum () const

double	Count () const

void	SetDim (int32 dim)

Public Member Functions inherited from Component
	Component ()

virtual std::string	Type () const =0

virtual int32	Index () const
	Returns the index in the sequence of layers in the neural net; intended only to be used in debugging information. More...

virtual void	SetIndex (int32 index)

virtual std::vector< int32 >	Context () const
	Return a vector describing the temporal context this component requires for each frame of output, as a sorted list. More...

virtual void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const =0
	Perform forward pass propagation Input->Output. More...

void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out) const
	A non-virtual propagate function that first resizes output if necessary. More...

virtual void	Backprop (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component to_update, CuMatrix< BaseFloat > in_deriv) const =0
	Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise). More...

virtual bool	BackpropNeedsInput () const

virtual bool	BackpropNeedsOutput () const

virtual Component *	Copy () const =0
	Copy component (deep copy). More...

virtual std::string	Info () const

virtual	~Component ()

Protected Member Functions
void	UpdateStats (const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > *deriv=NULL)

const NonlinearComponent &	operator= (const NonlinearComponent &other)

Protected Attributes
int32	dim_

CuVector< double >	value_sum_

CuVector< double >	deriv_sum_

double	count_

std::mutex	mutex_

Friends
class	NormalizationComponent

class	SigmoidComponent

class	TanhComponent

class	SoftmaxComponent

class	LogSoftmaxComponent

class	RectifiedLinearComponent

class	SoftHingeComponent

Additional Inherited Members
Static Public Member Functions inherited from Component
static Component *	ReadNew (std::istream &is, bool binary)
	Read component from stream. More...

static Component *	NewFromString (const std::string &initializer_line)
	Initialize the Component from one line that will contain first the type, e.g. More...

static Component *	NewComponentOfType (const std::string &type)
	Return a new Component of the given type e.g. More...

Detailed Description

This kind of Component is a base-class for things like sigmoid and softmax.

Definition at line 352 of file nnet-component.h.

Constructor & Destructor Documentation

◆ NonlinearComponent() [1/3]

NonlinearComponent ( int32 dim )

inlineexplicit

Definition at line 355 of file nnet-component.h.

355 { Init(dim); }

kaldi::nnet2::NonlinearComponent::Init

void Init(int32 dim)

Definition: nnet-component.h:354

◆ NonlinearComponent() [2/3]

NonlinearComponent ( )

inline

Definition at line 356 of file nnet-component.h.

356 : dim_(0) { } // e.g. prior to Read().

kaldi::nnet2::NonlinearComponent::dim_

int32 dim_

Definition: nnet-component.h:402

◆ NonlinearComponent() [3/3]

NonlinearComponent ( const NonlinearComponent & other )

explicit

Definition at line 405 of file nnet-component.cc.

                                                                      :
     dim_(other.dim_), value_sum_(other.value_sum_), deriv_sum_(other.deriv_sum_),
     count_(other.count_) { }

Member Function Documentation

◆ Add()

void Add	(	BaseFloat	alpha,
		const NonlinearComponent &	other
	)

Definition at line 362 of file nnet-component.cc.

References NonlinearComponent::count_, NonlinearComponent::deriv_sum_, CuVectorBase< Real >::Dim(), and NonlinearComponent::value_sum_.

Referenced by Nnet::AddNnet(), Nnet::CopyStatsFrom(), and main().

                                                                              {
   if (value_sum_.Dim() == 0 && other.value_sum_.Dim() != 0)
     value_sum_.Resize(other.value_sum_.Dim());
   if (deriv_sum_.Dim() == 0 && other.deriv_sum_.Dim() != 0)
     deriv_sum_.Resize(other.deriv_sum_.Dim());
   if (other.value_sum_.Dim() != 0)
     value_sum_.AddVec(alpha, other.value_sum_);
   if (other.deriv_sum_.Dim() != 0)
     deriv_sum_.AddVec(alpha, other.deriv_sum_);
   count_ += alpha * other.count_;
 }

◆ Count()

double Count ( ) const

inline

Definition at line 379 of file nnet-component.h.

Referenced by NnetStats::AddStatsFromNnet(), and kaldi::nnet2::FixNnet().

379 { return count_; }

kaldi::nnet2::NonlinearComponent::count_

double count_

Definition: nnet-component.h:406

◆ DerivSum()

const CuVector<double>& DerivSum ( ) const

inline

Definition at line 378 of file nnet-component.h.

Referenced by NnetStats::AddStatsFromNnet(), and kaldi::nnet2::FixNnet().

378 { return deriv_sum_; }

kaldi::nnet2::NonlinearComponent::deriv_sum_

CuVector< double > deriv_sum_

Definition: nnet-component.h:404

◆ Init()

void Init ( int32 dim )

inline

Definition at line 354 of file nnet-component.h.

Referenced by NonlinearComponent::InitFromString(), MaxoutComponent::InitFromString(), PnormComponent::InitFromString(), PowerComponent::InitFromString(), and ScaleComponent::InitFromString().

354 { dim_ = dim; count_ = 0.0; }

kaldi::nnet2::NonlinearComponent::dim_

int32 dim_

Definition: nnet-component.h:402

kaldi::nnet2::NonlinearComponent::count_

double count_

Definition: nnet-component.h:406

◆ InitFromString()

void InitFromString ( std::string args )

virtual

We implement InitFromString at this level.

Implements Component.

Reimplemented in PowerComponent.

Definition at line 409 of file nnet-component.cc.

References NonlinearComponent::Init(), KALDI_ERR, kaldi::nnet2::ParseFromString(), and Component::Type().

Referenced by kaldi::nnet2::UnitTestSigmoidComponent().

                                                       {
   std::string orig_args(args);
   int32 dim;
   bool ok = ParseFromString("dim", &args, &dim);
   if (!ok || !args.empty() || dim <= 0)
     KALDI_ERR << "Invalid initializer for layer of type "
               << Type() << ": \"" << orig_args << "\"";
   Init(dim);
 }

◆ InputDim()

virtual int32 InputDim ( ) const

inlinevirtual

Get size of input vectors.

Implements Component.

Reimplemented in PowerComponent.

Definition at line 359 of file nnet-component.h.

Referenced by kaldi::nnet2::FixNnet().

359 { return dim_; }

kaldi::nnet2::NonlinearComponent::dim_

int32 dim_

Definition: nnet-component.h:402

◆ operator=()

const NonlinearComponent& operator= ( const NonlinearComponent & other )

protected

◆ OutputDim()

virtual int32 OutputDim ( ) const

inlinevirtual

Get size of output vectors.

Implements Component.

Reimplemented in PowerComponent.

Definition at line 360 of file nnet-component.h.

360 { return dim_; }

kaldi::nnet2::NonlinearComponent::dim_

int32 dim_

Definition: nnet-component.h:402

◆ Read()

void Read	(	std::istream &	is,
		bool	binary
	)

virtual

We implement Read at this level as it just needs the Type().

Implements Component.

Reimplemented in PowerComponent.

Definition at line 374 of file nnet-component.cc.

References kaldi::nnet2::ExpectOneOrTwoTokens(), kaldi::ExpectToken(), kaldi::ReadBasicType(), and Component::Type().

                                                          {
   std::ostringstream ostr_beg, ostr_end;
   ostr_beg << "<" << Type() << ">"; // e.g. "<SigmoidComponent>"
   ostr_end << "</" << Type() << ">"; // e.g. "</SigmoidComponent>"
   ExpectOneOrTwoTokens(is, binary, ostr_beg.str(), "<Dim>");
   ReadBasicType(is, binary, &dim_); // Read dimension.
   ExpectToken(is, binary, "<ValueSum>");
   value_sum_.Read(is, binary);
   ExpectToken(is, binary, "<DerivSum>");
   deriv_sum_.Read(is, binary);
   ExpectToken(is, binary, "<Count>");
   ReadBasicType(is, binary, &count_);
   ExpectToken(is, binary, ostr_end.str());
 }

◆ Scale()

void Scale ( BaseFloat scale )

Definition at line 356 of file nnet-component.cc.

Referenced by Nnet::AddNnet(), Nnet::CopyStatsFrom(), main(), Nnet::Scale(), Nnet::SetZero(), and Nnet::ZeroStats().

                                               {
   value_sum_.Scale(scale);
   deriv_sum_.Scale(scale);
   count_ *= scale;
 }

◆ SetDim()

void SetDim ( int32 dim )

Definition at line 321 of file nnet-component.cc.

References KALDI_ASSERT.

                                          {
   KALDI_ASSERT(dim > 0);
   dim_ = dim;
   value_sum_.Resize(dim);
   deriv_sum_.Resize(dim);
   count_ = 0.0;
 }

◆ UpdateStats()

void UpdateStats	(	const CuMatrixBase< BaseFloat > &	out_value,
		const CuMatrixBase< BaseFloat > *	deriv = `NULL`
	)

protected

Definition at line 329 of file nnet-component.cc.

References CuVectorBase< Real >::AddRowSumMat(), CuMatrixBase< Real >::Dim(), Component::InputDim(), KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

Referenced by SigmoidComponent::Backprop(), TanhComponent::Backprop(), RectifiedLinearComponent::Backprop(), SoftHingeComponent::Backprop(), SoftmaxComponent::Backprop(), and LogSoftmaxComponent::Backprop().

                                                                            {
   KALDI_ASSERT(out_value.NumCols() == InputDim());
   // Check we have the correct dimensions.
   if (value_sum_.Dim() != InputDim() ||
       (deriv != NULL && deriv_sum_.Dim() != InputDim())) {
     std::lock_guard<std::mutex> lock(mutex_);
     if (value_sum_.Dim() != InputDim()) {
       value_sum_.Resize(InputDim());
       count_ = 0.0;
     }
     if (deriv != NULL && deriv_sum_.Dim() != InputDim()) {
       deriv_sum_.Resize(InputDim());
       count_ = 0.0;
       value_sum_.SetZero();
     }
   }
   count_ += out_value.NumRows();
   CuVector<BaseFloat> temp(InputDim());
   temp.AddRowSumMat(1.0, out_value, 0.0);
   value_sum_.AddVec(1.0, temp);
   if (deriv != NULL) {
     temp.AddRowSumMat(1.0, *deriv, 0.0);
     deriv_sum_.AddVec(1.0, temp);
   }
 }

◆ ValueSum()

const CuVector<double>& ValueSum ( ) const

inline

Definition at line 377 of file nnet-component.h.

Referenced by NnetStats::AddStatsFromNnet().

377 { return value_sum_; }

kaldi::nnet2::NonlinearComponent::value_sum_

CuVector< double > value_sum_

Definition: nnet-component.h:403

◆ Write()

void Write	(	std::ostream &	os,
		bool	binary
	)		const

virtual

Write component to stream.

Implements Component.

Reimplemented in PowerComponent.

Definition at line 389 of file nnet-component.cc.

References Component::Type(), kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                                 {
   std::ostringstream ostr_beg, ostr_end;
   ostr_beg << "<" << Type() << ">"; // e.g. "<SigmoidComponent>"
   ostr_end << "</" << Type() << ">"; // e.g. "</SigmoidComponent>"
   WriteToken(os, binary, ostr_beg.str());
   WriteToken(os, binary, "<Dim>");
   WriteBasicType(os, binary, dim_);
   WriteToken(os, binary, "<ValueSum>");
   value_sum_.Write(os, binary);
   WriteToken(os, binary, "<DerivSum>");
   deriv_sum_.Write(os, binary);
   WriteToken(os, binary, "<Count>");
   WriteBasicType(os, binary, count_);
   WriteToken(os, binary, ostr_end.str());
 }

Friends And Related Function Documentation

◆ LogSoftmaxComponent

friend class LogSoftmaxComponent

friend

Definition at line 389 of file nnet-component.h.

◆ NormalizationComponent

friend class NormalizationComponent

friend

Definition at line 385 of file nnet-component.h.

◆ RectifiedLinearComponent

friend class RectifiedLinearComponent

friend

Definition at line 390 of file nnet-component.h.

◆ SigmoidComponent

friend class SigmoidComponent

friend

Definition at line 386 of file nnet-component.h.

◆ SoftHingeComponent

friend class SoftHingeComponent

friend

Definition at line 391 of file nnet-component.h.

◆ SoftmaxComponent

friend class SoftmaxComponent

friend

Definition at line 388 of file nnet-component.h.

◆ TanhComponent

friend class TanhComponent

friend

Definition at line 387 of file nnet-component.h.

Member Data Documentation

◆ count_

double count_

protected

Definition at line 406 of file nnet-component.h.

Referenced by NonlinearComponent::Add(), and SoftmaxComponent::MixUp().

◆ deriv_sum_

CuVector<double> deriv_sum_

protected

Definition at line 404 of file nnet-component.h.

Referenced by NonlinearComponent::Add().

◆ dim_

int32 dim_

protected

Definition at line 402 of file nnet-component.h.

Referenced by PowerComponent::Info(), ScaleComponent::Info(), PowerComponent::Init(), ScaleComponent::Init(), SoftmaxComponent::MixUp(), PowerComponent::Read(), ScaleComponent::Read(), PowerComponent::Write(), and ScaleComponent::Write().

◆ mutex_

std::mutex mutex_

protected

Definition at line 408 of file nnet-component.h.

◆ value_sum_

CuVector<double> value_sum_

protected

Definition at line 403 of file nnet-component.h.

Referenced by NonlinearComponent::Add(), and SoftmaxComponent::MixUp().

The documentation for this class was generated from the following files:

nnet2/nnet-component.h
nnet2/nnet-component.cc

Public Member Functions

Protected Member Functions

Protected Attributes

Friends

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ NonlinearComponent() [1/3]

◆ NonlinearComponent() [2/3]

◆ NonlinearComponent() [3/3]

Member Function Documentation

◆ Add()

◆ Count()

◆ DerivSum()

◆ Init()

◆ InitFromString()

◆ InputDim()

◆ operator=()

◆ OutputDim()

◆ Read()

◆ Scale()

◆ SetDim()

◆ UpdateStats()

◆ ValueSum()

◆ Write()

Friends And Related Function Documentation

◆ LogSoftmaxComponent

◆ NormalizationComponent

◆ RectifiedLinearComponent

◆ SigmoidComponent

◆ SoftHingeComponent

◆ SoftmaxComponent

◆ TanhComponent

Member Data Documentation

◆ count_

◆ deriv_sum_

◆ dim_

◆ mutex_

◆ value_sum_