NonlinearComponent Class Reference

#include <nnet-component-itf.h>

Inheritance diagram for NonlinearComponent:

Collaboration diagram for NonlinearComponent:

Public Member Functions
	NonlinearComponent ()
	NonlinearComponent (const NonlinearComponent &other)
virtual int32	InputDim () const
	Returns input-dimension of this component. More...
virtual int32	OutputDim () const
	Returns output-dimension of this component. More...
virtual void	InitFromConfig (ConfigLine *cfl)
	Initialize, from a ConfigLine object. More...
virtual void	Read (std::istream &is, bool binary)
	We implement Read at this level as it just needs the Type(). More...
virtual void	ZeroStats ()
	Components that provide an implementation of StoreStats should also provide an implementation of ZeroStats(), to set those stats to zero. More...
virtual std::string	Info () const
	Returns some text-form information about this component, for diagnostics. More...
virtual void	Write (std::ostream &os, bool binary) const
	Write component to stream. More...
virtual void	Scale (BaseFloat scale)
	This virtual function when called on – an UpdatableComponent scales the parameters by "scale" when called by an UpdatableComponent. More...
virtual void	Add (BaseFloat alpha, const Component &other)
	This virtual function when called by – an UpdatableComponent adds the parameters of another updatable component, times some constant, to the current parameters. More...
virtual void	ConsolidateMemory ()
	This virtual function relates to memory management, and avoiding fragmentation. More...
const CuVector< double > &	ValueSum () const
const CuVector< double > &	DerivSum () const
double	Count () const
Public Member Functions inherited from Component
virtual void *	Propagate (const ComponentPrecomputedIndexes *indexes, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const =0
	Propagate function. More...
virtual void	Backprop (const std::string &debug_info, const ComponentPrecomputedIndexes *indexes, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, void *memo, Component *to_update, CuMatrixBase< BaseFloat > *in_deriv) const =0
	Backprop function; depending on which of the arguments 'to_update' and 'in_deriv' are non-NULL, this can compute input-data derivatives and/or perform model update. More...
virtual void	StoreStats (const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, void *memo)
	This function may store stats on average activation values, and for some component types, the average value of the derivative of the nonlinearity. More...
virtual void	GetInputIndexes (const MiscComputationInfo &misc_info, const Index &output_index, std::vector< Index > *desired_indexes) const
	This function only does something interesting for non-simple Components. More...
virtual bool	IsComputable (const MiscComputationInfo &misc_info, const Index &output_index, const IndexSet &input_index_set, std::vector< Index > *used_inputs) const
	This function only does something interesting for non-simple Components, and it exists to make it possible to manage optionally-required inputs. More...
virtual void	ReorderIndexes (std::vector< Index > *input_indexes, std::vector< Index > *output_indexes) const
	This function only does something interesting for non-simple Components. More...
virtual ComponentPrecomputedIndexes *	PrecomputeIndexes (const MiscComputationInfo &misc_info, const std::vector< Index > &input_indexes, const std::vector< Index > &output_indexes, bool need_backprop) const
	This function must return NULL for simple Components. More...
virtual std::string	Type () const =0
	Returns a string such as "SigmoidComponent", describing the type of the object. More...
virtual int32	Properties () const =0
	Return bitmask of the component's properties. More...
virtual Component *	Copy () const =0
	Copies component (deep copy). More...
virtual void	DeleteMemo (void *memo) const
	This virtual function only needs to be overwritten by Components that return a non-NULL memo from their Propagate() function. More...
	Component ()
virtual	~Component ()

Protected Types
enum	{ kUnsetThreshold = -1000 }

Protected Member Functions
void	StoreStatsInternal (const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > *deriv=NULL)
void	StoreBackpropStats (const CuMatrixBase< BaseFloat > &out_deriv)
const NonlinearComponent &	operator= (const NonlinearComponent &other)

Protected Attributes
int32	dim_
int32	block_dim_
CuVector< double >	value_sum_
CuVector< double >	deriv_sum_
double	count_
CuVector< double >	oderiv_sumsq_
double	oderiv_count_
double	num_dims_self_repaired_
double	num_dims_processed_
BaseFloat	self_repair_lower_threshold_
BaseFloat	self_repair_upper_threshold_
BaseFloat	self_repair_scale_

Friends
class	SigmoidComponent
class	TanhComponent
class	SoftmaxComponent
class	LogSoftmaxComponent
class	RectifiedLinearComponent

Additional Inherited Members
Static Public Member Functions inherited from Component
static Component *	ReadNew (std::istream &is, bool binary)
	Read component from stream (works out its type). Dies on error. More...
static Component *	NewComponentOfType (const std::string &type)
	Returns a new Component of the given type e.g. More...

Detailed Description

Definition at line 613 of file nnet-component-itf.h.

Member Enumeration Documentation

anonymous enum

anonymous enum

protected

Enumerator
kUnsetThreshold

Definition at line 650 of file nnet-component-itf.h.

{ kUnsetThreshold = -1000 };

Constructor & Destructor Documentation

NonlinearComponent() [1/2]

NonlinearComponent

(

)

Definition at line 605 of file nnet-component-itf.cc.

                                      :
    dim_(-1), block_dim_(-1), count_(0.0), oderiv_count_(0.0),
    num_dims_self_repaired_(0.0), num_dims_processed_(0.0),
    self_repair_lower_threshold_(kUnsetThreshold),
    self_repair_upper_threshold_(kUnsetThreshold),
    self_repair_scale_(0.0) { }

NonlinearComponent() [2/2]

NonlinearComponent ( const NonlinearComponent &  other )

explicit

Definition at line 612 of file nnet-component-itf.cc.

                                                                     :
    dim_(other.dim_), block_dim_(other.block_dim_),
    value_sum_(other.value_sum_), deriv_sum_(other.deriv_sum_),
    count_(other.count_), oderiv_sumsq_(other.oderiv_sumsq_),
    oderiv_count_(other.oderiv_count_),
    num_dims_self_repaired_(other.num_dims_self_repaired_),
    num_dims_processed_(other.num_dims_processed_),
    self_repair_lower_threshold_(other.self_repair_lower_threshold_),
    self_repair_upper_threshold_(other.self_repair_upper_threshold_),
    self_repair_scale_(other.self_repair_scale_) { }

Member Function Documentation

Add()

void Add ( BaseFloat  alpha, const Component &  other  )

virtual

This virtual function when called by – an UpdatableComponent adds the parameters of another updatable component, times some constant, to the current parameters.

– a NonlinearComponent (or another component that stores stats, like BatchNormComponent)– it relates to adding stats. Otherwise it will normally do nothing.

Reimplemented from Component.

Definition at line 459 of file nnet-component-itf.cc.

References NonlinearComponent::count_, NonlinearComponent::deriv_sum_, CuVectorBase< Real >::Dim(), KALDI_ASSERT, NonlinearComponent::num_dims_processed_, NonlinearComponent::num_dims_self_repaired_, NonlinearComponent::oderiv_count_, NonlinearComponent::oderiv_sumsq_, and NonlinearComponent::value_sum_.

                                                                       {
  const NonlinearComponent *other =
      dynamic_cast<const NonlinearComponent*>(&other_in);
  KALDI_ASSERT(other != NULL);
  if (value_sum_.Dim() == 0 && other->value_sum_.Dim() != 0)
    value_sum_.Resize(other->value_sum_.Dim());
  if (deriv_sum_.Dim() == 0 && other->deriv_sum_.Dim() != 0)
    deriv_sum_.Resize(other->deriv_sum_.Dim());
  if (oderiv_sumsq_.Dim() == 0 && other->oderiv_sumsq_.Dim() != 0)
    oderiv_sumsq_.Resize(other->oderiv_sumsq_.Dim());
  if (other->value_sum_.Dim() != 0)
    value_sum_.AddVec(alpha, other->value_sum_);
  if (other->deriv_sum_.Dim() != 0)
    deriv_sum_.AddVec(alpha, other->deriv_sum_);
  if (other->oderiv_sumsq_.Dim() != 0)
    oderiv_sumsq_.AddVec(alpha, other->oderiv_sumsq_);
  count_ += alpha * other->count_;
  oderiv_count_ += alpha * other->oderiv_count_;
  num_dims_self_repaired_ += alpha * other->num_dims_self_repaired_;
  num_dims_processed_ += alpha * other->num_dims_processed_;
}

ConsolidateMemory()

void ConsolidateMemory ( )

virtual

This virtual function relates to memory management, and avoiding fragmentation.

It is called only once per model, after we do the first minibatch of training. The default implementation does nothing, but it can be overridden by child classes, where it may re-initialize certain quantities that may possibly have been allocated during the forward pass (e.g. certain statistics; OnlineNaturalGradient objects). We use our own CPU-based allocator (see cu-allocator.h) and since it can't do paging since we're not in control of the GPU page table, fragmentation can be a problem. The allocator always tries to put things in 'low-address memory' (i.e. at smaller memory addresses) near the beginning of the block it allocated, to avoid fragmentation; but if permanent things (belonging to the model) are allocated in the forward pass, they can permanently stay in high memory. This function helps to prevent that, by re-allocating those things into low-address memory (It's important that it's called after all the temporary buffers for the forward-backward have been freed, so that there is low-address memory available)).

Reimplemented from Component.

Definition at line 636 of file nnet-component-itf.cc.

References NonlinearComponent::deriv_sum_, NonlinearComponent::oderiv_sumsq_, CuVector< Real >::Swap(), and NonlinearComponent::value_sum_.

                                           {
  { CuVector<double> temp(value_sum_); value_sum_.Swap(&temp); }
  { CuVector<double> temp(deriv_sum_); deriv_sum_.Swap(&temp); }
  { CuVector<double> temp(oderiv_sumsq_); oderiv_sumsq_.Swap(&temp); }
}

Count()

double Count ( ) const

inline

Definition at line 647 of file nnet-component-itf.h.

{ return count_; }

DerivSum()

const CuVector<double>& DerivSum ( ) const

inline

Definition at line 645 of file nnet-component-itf.h.

{ return deriv_sum_; }

Info()

std::string Info ( ) const

virtual

Returns some text-form information about this component, for diagnostics.

Starts with the type of the component. E.g. "SigmoidComponent dim=900", although most components will have much more info.

Reimplemented from Component.

Definition at line 409 of file nnet-component-itf.cc.

References VectorBase< Real >::ApplyFloor(), VectorBase< Real >::ApplyPow(), VectorBase< Real >::Scale(), kaldi::nnet3::SummarizeVector(), and Component::Type().

                                         {
  std::stringstream stream;
  stream << Type() << ", dim=" << dim_;
  if (block_dim_ != dim_)
    stream << ", block-dim=" << block_dim_;
  if (self_repair_lower_threshold_ != BaseFloat(kUnsetThreshold))
    stream << ", self-repair-lower-threshold=" << self_repair_lower_threshold_;
  if (self_repair_upper_threshold_ != BaseFloat(kUnsetThreshold))
    stream << ", self-repair-upper-threshold=" << self_repair_upper_threshold_;
  if (self_repair_scale_ != 0.0)
    stream << ", self-repair-scale=" << self_repair_scale_;
  if (count_ > 0 && value_sum_.Dim() == dim_) {
    stream << ", count=" << std::setprecision(3) << count_
           << std::setprecision(6);
    stream << ", self-repaired-proportion="
           << (num_dims_processed_ > 0 ?
               num_dims_self_repaired_ / num_dims_processed_ : 0);
    Vector<double> value_avg_dbl(value_sum_);
    Vector<BaseFloat> value_avg(value_avg_dbl);
    value_avg.Scale(1.0 / count_);
    stream << ", value-avg=" << SummarizeVector(value_avg);
    if (deriv_sum_.Dim() == dim_) {
      Vector<double> deriv_avg(deriv_sum_);
      deriv_avg.Scale(1.0 / count_);
      stream << ", deriv-avg=" << SummarizeVector(deriv_avg);
    }
  }
  if (oderiv_count_ > 0 && oderiv_sumsq_.Dim() == dim_) {
    Vector<double> oderiv_rms(oderiv_sumsq_);
    oderiv_rms.Scale(1.0 / oderiv_count_);
    // The ApplyMin() is so that the statement after it does not fail even if we
    // had subtracted models (e.g. in full_progress.*.log).
    oderiv_rms.ApplyFloor(0.0);
    oderiv_rms.ApplyPow(0.5);
    stream << ", oderiv-rms=" << SummarizeVector(oderiv_rms)
           << ", oderiv-count=" << oderiv_count_;
  }
  return stream.str();
}

InitFromConfig()

void InitFromConfig ( ConfigLine *  cfl )

virtual

Initialize, from a ConfigLine object.

Parameters

[in]

cfl

A ConfigLine containing any parameters that are needed for initialization. For example: "dim=100 param-stddev=0.1"

Implements Component.

Definition at line 623 of file nnet-component-itf.cc.

References NonlinearComponent::block_dim_, NonlinearComponent::dim_, ConfigLine::GetValue(), ConfigLine::HasUnusedValues(), KALDI_ERR, NonlinearComponent::self_repair_lower_threshold_, NonlinearComponent::self_repair_scale_, NonlinearComponent::self_repair_upper_threshold_, Component::Type(), and ConfigLine::WholeLine().

                                                       {
  bool ok = cfl->GetValue("dim", &dim_);
  block_dim_ = dim_;
  cfl->GetValue("block-dim", &block_dim_);
  cfl->GetValue("self-repair-lower-threshold", &self_repair_lower_threshold_);
  cfl->GetValue("self-repair-upper-threshold", &self_repair_upper_threshold_);
  cfl->GetValue("self-repair-scale", &self_repair_scale_);
  if (!ok || cfl->HasUnusedValues() || dim_ <= 0 ||
      block_dim_ <= 0 || dim_ % block_dim_ != 0)
    KALDI_ERR << "Invalid initializer for layer of type "
              << Type() << ": \"" << cfl->WholeLine() << "\"";
}

InputDim()

virtual int32 InputDim ( ) const

inlinevirtual

Returns input-dimension of this component.

Implements Component.

Definition at line 619 of file nnet-component-itf.h.

{ return dim_; }

operator=()

const NonlinearComponent& operator= ( const NonlinearComponent &  other )

protected

OutputDim()

virtual int32 OutputDim ( ) const

inlinevirtual

Returns output-dimension of this component.

Implements Component.

Definition at line 620 of file nnet-component-itf.h.

References kaldi::nnet3::ConsolidateMemory(), ComponentPrecomputedIndexes::Read(), and ComponentPrecomputedIndexes::Write().

{ return dim_; }

Read()

void Read ( std::istream &  is, bool  binary  )

virtual

We implement Read at this level as it just needs the Type().

Implements Component.

Definition at line 481 of file nnet-component-itf.cc.

References kaldi::ExpectOneOrTwoTokens(), kaldi::nnet3::ExpectToken(), KALDI_ERR, kaldi::PeekToken(), kaldi::ReadBasicType(), kaldi::ReadToken(), and Component::Type().

                                                         {
  std::ostringstream ostr_beg, ostr_end;
  ostr_beg << "<" << Type() << ">"; // e.g. "<SigmoidComponent>"
  ostr_end << "</" << Type() << ">"; // e.g. "</SigmoidComponent>"
  ExpectOneOrTwoTokens(is, binary, ostr_beg.str(), "<Dim>");
  ReadBasicType(is, binary, &dim_); // Read dimension.
  if (PeekToken(is, binary) == 'B') {
    ExpectToken(is, binary, "<BlockDim>");
    ReadBasicType(is, binary, &block_dim_);
  } else {
    block_dim_ = dim_;
  }
  ExpectToken(is, binary, "<ValueAvg>");
  value_sum_.Read(is, binary);
  ExpectToken(is, binary, "<DerivAvg>");
  deriv_sum_.Read(is, binary);
  ExpectToken(is, binary, "<Count>");
  ReadBasicType(is, binary, &count_);
  if (PeekToken(is, binary) == 'O') {
    ExpectToken(is, binary, "<OderivRms>");
    oderiv_sumsq_.Read(is, binary);
    oderiv_sumsq_.ApplyPow(2.0);
    ExpectToken(is, binary, "<OderivCount>");
    ReadBasicType(is, binary, &oderiv_count_);
  } else {
    oderiv_count_ = 0.0;
    oderiv_sumsq_.Resize(0);
  }
  value_sum_.Scale(count_);
  deriv_sum_.Scale(count_);
  oderiv_sumsq_.Scale(oderiv_count_);

  std::string token;
  ReadToken(is, binary, &token);
  if (token[0] != '<') {
    // this should happen only rarely, in case we couldn't push back the
    // '<' to the stream in PeekToken().
    token = '<' + token;
  }
  if (token == "<NumDimsSelfRepaired>") {
    ReadBasicType(is, binary, &num_dims_self_repaired_);
    ReadToken(is, binary, &token);
  }
  if (token == "<NumDimsProcessed>") {
    ReadBasicType(is, binary, &num_dims_processed_);
    ReadToken(is, binary, &token);
  }
  if (token == "<SelfRepairLowerThreshold>") {
    ReadBasicType(is, binary, &self_repair_lower_threshold_);
    ReadToken(is, binary, &token);
  }
  if (token == "<SelfRepairUpperThreshold>") {
    ReadBasicType(is, binary, &self_repair_upper_threshold_);
    ReadToken(is, binary, &token);
  }
  if (token == "<SelfRepairScale>") {
    ReadBasicType(is, binary, &self_repair_scale_);
    ReadToken(is, binary, &token);
  }
  if (token != ostr_end.str()) {
    KALDI_ERR << "Expected token " << ostr_end.str()
              << ", got " << token;
  }
}

Scale()

void Scale ( BaseFloat  scale )

virtual

This virtual function when called on – an UpdatableComponent scales the parameters by "scale" when called by an UpdatableComponent.

– a Nonlinear component (or another component that stores stats, like BatchNormComponent)– it relates to scaling activation stats, not parameters. Otherwise it will normally do nothing.

Reimplemented from Component.

Definition at line 449 of file nnet-component-itf.cc.

                                              {
  value_sum_.Scale(scale);
  deriv_sum_.Scale(scale);
  oderiv_sumsq_.Scale(scale);
  count_ *= scale;
  oderiv_count_ *= scale;
  num_dims_self_repaired_ *= scale;
  num_dims_processed_ *= scale;
}

StoreBackpropStats()

void StoreBackpropStats ( const CuMatrixBase< BaseFloat > &  out_deriv )

protected

Definition at line 377 of file nnet-component-itf.cc.

References CuVectorBase< Real >::AddDiagMat2(), KALDI_ASSERT, kaldi::kTrans, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), and kaldi::RandInt().

Referenced by SigmoidComponent::Backprop(), TanhComponent::Backprop(), RectifiedLinearComponent::Backprop(), SoftmaxComponent::Backprop(), and LogSoftmaxComponent::Backprop().

                                              {
  // Only store these stats about every 4 minibatches.  Make sure to always
  // store the stats on the very first minibatch, or it would interact badly
  // with the ConsolidateMemory() code.
  if (RandInt(0, 3) == 0 && oderiv_count_ != 0)
    return;

  KALDI_ASSERT(out_deriv.NumCols() == dim_);

  // Check we have the correct dimensions.
  if (oderiv_sumsq_.Dim() != dim_) {
    oderiv_sumsq_.Resize(dim_);
    oderiv_count_ = 0.0;
  }
  CuVector<BaseFloat> temp(dim_);
  temp.AddDiagMat2(1.0, out_deriv, kTrans, 0.0);
  oderiv_sumsq_.AddVec(1.0, temp);
  oderiv_count_ += out_deriv.NumRows();
}

StoreStatsInternal()

void StoreStatsInternal ( const CuMatrixBase< BaseFloat > &  out_value, const CuMatrixBase< BaseFloat > *  deriv = NULL  )

protected

Definition at line 349 of file nnet-component-itf.cc.

References CuVectorBase< Real >::AddRowSumMat(), CuMatrixBase< Real >::Dim(), KALDI_ASSERT, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

                                          {
  KALDI_ASSERT(out_value.NumCols() == dim_);

  // Check we have the correct dimensions.
  if (value_sum_.Dim() != dim_ ||
      (deriv != NULL && deriv_sum_.Dim() != dim_)) {
    if (value_sum_.Dim() != dim_) {
      value_sum_.Resize(dim_);
      count_ = 0.0;
    }
    if (deriv != NULL && deriv_sum_.Dim() != dim_) {
      deriv_sum_.Resize(dim_);
      count_ = 0.0;
      value_sum_.SetZero();
    }
  }
  count_ += out_value.NumRows();
  CuVector<BaseFloat> temp(dim_);
  temp.AddRowSumMat(1.0, out_value, 0.0);
  value_sum_.AddVec(1.0, temp);
  if (deriv != NULL) {
    temp.AddRowSumMat(1.0, *deriv, 0.0);
    deriv_sum_.AddVec(1.0, temp);
  }
}

ValueSum()

const CuVector<double>& ValueSum ( ) const

inline

Definition at line 644 of file nnet-component-itf.h.

{ return value_sum_; }

Write()

void Write ( std::ostream &  os, bool  binary  ) const

virtual

Write component to stream.

Implements Component.

Definition at line 546 of file nnet-component-itf.cc.

References VectorBase< Real >::ApplyFloor(), VectorBase< Real >::ApplyPow(), VectorBase< Real >::CopyFromVec(), Vector< Real >::Resize(), VectorBase< Real >::Scale(), Component::Type(), VectorBase< Real >::Write(), kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                                {
  std::ostringstream ostr_beg, ostr_end;
  ostr_beg << "<" << Type() << ">"; // e.g. "<SigmoidComponent>"
  ostr_end << "</" << Type() << ">"; // e.g. "</SigmoidComponent>"
  WriteToken(os, binary, ostr_beg.str());
  WriteToken(os, binary, "<Dim>");
  WriteBasicType(os, binary, dim_);
  if (block_dim_ != dim_) {
    WriteToken(os, binary, "<BlockDim>");
    WriteBasicType(os, binary, block_dim_);
  }
  // Write the values and derivatives in a count-normalized way, for
  // greater readability in text form.
  WriteToken(os, binary, "<ValueAvg>");
  Vector<BaseFloat> temp(value_sum_);
  if (count_ != 0.0) temp.Scale(1.0 / count_);
  temp.Write(os, binary);

  WriteToken(os, binary, "<DerivAvg>");
  temp.Resize(deriv_sum_.Dim());
  temp.CopyFromVec(deriv_sum_);
  if (count_ != 0.0) temp.Scale(1.0 / count_);
  temp.Write(os, binary);

  WriteToken(os, binary, "<Count>");
  WriteBasicType(os, binary, count_);

  WriteToken(os, binary, "<OderivRms>");
  temp.Resize(oderiv_sumsq_.Dim());
  temp.CopyFromVec(oderiv_sumsq_);
  if (oderiv_count_ != 0.0) temp.Scale(1.0 / oderiv_count_);
  // The ApplyMin() is so that the statement after it does not fail even if we
  // had subtracted models (e.g. in full_progress.*.log).
  temp.ApplyFloor(0.0);
  temp.ApplyPow(0.5);
  temp.Write(os, binary);

  WriteToken(os, binary, "<OderivCount>");
  WriteBasicType(os, binary, oderiv_count_);

  WriteToken(os, binary, "<NumDimsSelfRepaired>");
  WriteBasicType(os, binary, num_dims_self_repaired_);
  WriteToken(os, binary, "<NumDimsProcessed>");
  WriteBasicType(os, binary, num_dims_processed_);
  if (self_repair_lower_threshold_ != kUnsetThreshold) {
    WriteToken(os, binary, "<SelfRepairLowerThreshold>");
    WriteBasicType(os, binary, self_repair_lower_threshold_);
  }
  if (self_repair_upper_threshold_ != kUnsetThreshold) {
    WriteToken(os, binary, "<SelfRepairUpperThreshold>");
    WriteBasicType(os, binary, self_repair_upper_threshold_);
  }
  if (self_repair_scale_ != 0.0) {
    WriteToken(os, binary, "<SelfRepairScale>");
    WriteBasicType(os, binary, self_repair_scale_);
  }
  WriteToken(os, binary, ostr_end.str());
}

ZeroStats()

void ZeroStats ( )

virtual

Components that provide an implementation of StoreStats should also provide an implementation of ZeroStats(), to set those stats to zero.

Other components that store other types of statistics (e.g. regarding gradient clipping) should implement ZeroStats() also.

Reimplemented from Component.

Definition at line 399 of file nnet-component-itf.cc.

                                   {
  value_sum_.SetZero();
  deriv_sum_.SetZero();
  oderiv_sumsq_.SetZero();
  count_ = 0.0;
  oderiv_count_ = 0.0;
  num_dims_self_repaired_ = 0.0;
  num_dims_processed_ = 0.0;
}

Friends And Related Function Documentation

LogSoftmaxComponent

friend class LogSoftmaxComponent

friend

Definition at line 655 of file nnet-component-itf.h.

RectifiedLinearComponent

friend class RectifiedLinearComponent

friend

Definition at line 656 of file nnet-component-itf.h.

SigmoidComponent

friend class SigmoidComponent

friend

Definition at line 652 of file nnet-component-itf.h.

SoftmaxComponent

friend class SoftmaxComponent

friend

Definition at line 654 of file nnet-component-itf.h.

TanhComponent

friend class TanhComponent

friend

Definition at line 653 of file nnet-component-itf.h.

Member Data Documentation

block_dim_

int32 block_dim_

protected

Definition at line 678 of file nnet-component-itf.h.

Referenced by NonlinearComponent::InitFromConfig().

count_

double count_

protected

Definition at line 684 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add().

deriv_sum_

CuVector<double> deriv_sum_

protected

Definition at line 680 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add(), and NonlinearComponent::ConsolidateMemory().

dim_

int32 dim_

protected

Definition at line 673 of file nnet-component-itf.h.

Referenced by NonlinearComponent::InitFromConfig().

num_dims_processed_

double num_dims_processed_

protected

Definition at line 695 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add(), SigmoidComponent::RepairGradients(), TanhComponent::RepairGradients(), and RectifiedLinearComponent::RepairGradients().

num_dims_self_repaired_

double num_dims_self_repaired_

protected

Definition at line 694 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add(), SigmoidComponent::RepairGradients(), TanhComponent::RepairGradients(), and RectifiedLinearComponent::RepairGradients().

oderiv_count_

double oderiv_count_

protected

Definition at line 691 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add().

oderiv_sumsq_

CuVector<double> oderiv_sumsq_

protected

Definition at line 686 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add(), and NonlinearComponent::ConsolidateMemory().

self_repair_lower_threshold_

BaseFloat self_repair_lower_threshold_

protected

Definition at line 698 of file nnet-component-itf.h.

Referenced by NonlinearComponent::InitFromConfig().

self_repair_scale_

BaseFloat self_repair_scale_

protected

Definition at line 700 of file nnet-component-itf.h.

Referenced by NonlinearComponent::InitFromConfig().

self_repair_upper_threshold_

BaseFloat self_repair_upper_threshold_

protected

Definition at line 699 of file nnet-component-itf.h.

Referenced by NonlinearComponent::InitFromConfig().

value_sum_

CuVector<double> value_sum_

protected

Definition at line 679 of file nnet-component-itf.h.

Referenced by NonlinearComponent::Add(), and NonlinearComponent::ConsolidateMemory().

---

The documentation for this class was generated from the following files:

• nnet3/nnet-component-itf.h
• nnet3/nnet-component-itf.cc