Nnet Class Reference

#include <nnet-nnet.h>

Collaboration diagram for Nnet:

Public Member Functions
int32	NumComponents () const
	Returns number of components– think of this as similar to # of layers, but e.g. More...
const Component &	GetComponent (int32 c) const
Component &	GetComponent (int32 c)
void	SetComponent (int32 c, Component *component)
	Sets the c'th component to "component", taking ownership of the pointer and deleting the corresponding one that we own. More...
int32	LeftContext () const
	Returns the left-context summed over all the Components... More...
int32	RightContext () const
	Returns the right-context summed over all the Components... More...
int32	OutputDim () const
	The output dimension of the network – typically the number of pdfs. More...
int32	InputDim () const
	Dimension of the input features, e.g. More...
void	ComputeChunkInfo (int32 input_chunk_size, int32 num_chunks, std::vector< ChunkInfo > *chunk_info_out) const
	Uses the output of the Context() functions of the network, to compute a vector of size NumComponents() + 1 indexed by component-index c, of the chunk-info at the input of each layer c, where the c+1'th element contains the chunk-info at the output of that layer. More...
void	ZeroStats ()
void	CopyStatsFrom (const Nnet &nnet)
	Copies only the statistics in layers of type NonlinearComponewnt, from this neural net, leaving everything else fixed. More...
int32	FirstUpdatableComponent () const
	Returns the index of the lowest-numbered component which is updatable, or NumComponents() if none are updatable. More...
int32	LastUpdatableComponent () const
	Returns the index of the highest-numbered component which is updatable, or -1 if none are updatable. More...
int32	NumUpdatableComponents () const
	Returns the number of updatable components. More...
void	ScaleComponents (const VectorBase< BaseFloat > &scales)
	Scales the parameters of each of the updatable components. More...
void	RemoveDropout ()
	Excise any components of type DropoutComponent or AdditiveNoiseComponent. More...
void	SetDropoutScale (BaseFloat scale)
	Calls SetDropoutScale for all the dropout nodes. More...
void	RemovePreconditioning ()
	Replace any components of type AffineComponentPreconditioned with components of type AffineComponent. More...
void	SwitchToOnlinePreconditioning (int32 rank_in, int32 rank_out, int32 update_period, BaseFloat num_samples_history, BaseFloat alpha)
	Replaces any components of type AffineComponent or derived classes, with components of type AffineComponentPreconditionedOnline. More...
void	AddNnet (const VectorBase< BaseFloat > &scales, const Nnet &other)
	For each updatatable component, adds to it the corresponding element of "other" times the appropriate element of "scales" (which has the same format as for ScaleComponents(), i.e. More...
void	Scale (BaseFloat scale)
	Scales all the Components with the same scale. More...
void	AddNnet (BaseFloat alpha, const Nnet &other)
	Adds to *this, the other neural net times the scale "alpha". More...
void	LimitRankOfLastLayer (int32 dimension)
	Turns the last affine layer into two layers of the same type, with a smaller dimension in between– we're keeping the top singular values of the matrix. More...
void	AddNnet (BaseFloat alpha, Nnet *other, BaseFloat beta)
	This version of AddNnet adds to *this, alpha times *other, and then scales *other by beta. More...
void	Resize (int32 num_components)
	Removes final components from the neural network (used for debugging). More...
void	Collapse (bool match_updatableness)
	Where possible, collapse multiple affine or linear components in a sequence into a single one by composing the transforms. More...
void	SetIndexes ()
	Sets the index_ values of the components. More...
	Nnet (const Nnet &other)
	Nnet (const Nnet &nnet1, const Nnet &nnet2)
	Nnet ()
Nnet &	operator= (const Nnet &other)
void	Init (std::istream &is)
	Initialize from config file. More...
void	Init (std::vector< Component *> *components)
	This Init method works from a vector of components. More...
void	Append (Component *new_component)
	Appends this component to the components already in the neural net. More...
virtual	~Nnet ()
std::string	Info () const
void	Destroy ()
void	Write (std::ostream &os, bool binary) const
void	Read (std::istream &is, bool binary)
void	SetZero (bool treat_as_gradient)
void	ResizeOutputLayer (int32 new_num_pdfs)
	This function is used when doing transfer learning to a new system. More...
void	ScaleLearningRates (BaseFloat factor)
	Scale all the learning rates in the neural net by this factor. More...
void	ScaleLearningRates (std::map< std::string, BaseFloat > scale_factors)
	Scale all the learning rates in the neural net by the factors indexed by the type of component. More...
void	SetLearningRates (BaseFloat learning_rates)
	Set all the learning rates in the neural net to this value. More...
void	SetLearningRates (const VectorBase< BaseFloat > &learning_rates)
	Set all the learning rates in the neural net to these values (one for each updatable layer). More...
void	GetLearningRates (VectorBase< BaseFloat > *learning_rates) const
	Get all the learning rates in the neural net (the output must have dim equal to NumUpdatableComponents()). More...
void	ComponentDotProducts (const Nnet &other, VectorBase< BaseFloat > *dot_prod) const
void	Check () const
void	ResetGenerators ()
virtual int32	GetParameterDim () const
virtual void	Vectorize (VectorBase< BaseFloat > *params) const
virtual void	UnVectorize (const VectorBase< BaseFloat > &params)

Private Attributes
std::vector< Component * >	components_

Friends
class	NnetUpdater
class	DecodableNnet

Detailed Description

Definition at line 63 of file nnet-nnet.h.

Constructor & Destructor Documentation

Nnet() [1/3]

Nnet

(

const Nnet &

other

)

Definition at line 226 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, and Nnet::SetIndexes().

                           : components_(other.components_.size()) {
  for (size_t i = 0; i < other.components_.size(); i++)
    components_[i] = other.components_[i]->Copy();
  SetIndexes();
  Check();
}

Nnet() [2/3]

Nnet	(	const Nnet &	nnet1,
		const Nnet &	nnet2
	)

Definition at line 233 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, Nnet::InputDim(), KALDI_ERR, Nnet::OutputDim(), and Nnet::SetIndexes().

                                                 {
  int32 dim1 = other1.OutputDim(), dim2 = other2.InputDim();
  if (dim1 != dim2)
    KALDI_ERR << "Concatenating neural nets: dimension mismatch "
              << dim1 << " vs. " << dim2;
  for (size_t i = 0; i < other1.components_.size(); i++)
    components_.push_back(other1.components_[i]->Copy());
  for (size_t i = 0; i < other2.components_.size(); i++)
    components_.push_back(other2.components_[i]->Copy());
  SetIndexes();
  Check();
}

Nnet() [3/3]

Nnet ( )

inline

Definition at line 203 of file nnet-nnet.h.

References Nnet::Append(), Nnet::Init(), and Nnet::operator=().

Referenced by kaldi::nnet2::GenRandomNnet(), and Nnet::NumComponents().

{}

~Nnet()

virtual ~Nnet ( )

inlinevirtual

Definition at line 225 of file nnet-nnet.h.

References Nnet::Check(), Nnet::ComponentDotProducts(), Nnet::Destroy(), Nnet::GetLearningRates(), Nnet::GetParameterDim(), Nnet::Info(), Nnet::Read(), Nnet::ResetGenerators(), Nnet::ResizeOutputLayer(), Nnet::ScaleLearningRates(), Nnet::SetLearningRates(), Nnet::SetZero(), Nnet::UnVectorize(), Nnet::Vectorize(), and Nnet::Write().

{ Destroy(); }

Member Function Documentation

AddNnet() [1/3]

void AddNnet	(	const VectorBase< BaseFloat > &	scales,
		const Nnet &	other
	)

For each updatatable component, adds to it the corresponding element of "other" times the appropriate element of "scales" (which has the same format as for ScaleComponents(), i.e.

one entry for each updatable component).

Definition at line 576 of file nnet-nnet.cc.

References UpdatableComponent::Add(), VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), and Nnet::NumUpdatableComponents().

Referenced by kaldi::nnet2::AddDirection(), kaldi::nnet2::CombineNnets(), FastNnetCombiner::CombineNnets(), kaldi::nnet2::GetUpdateDirection(), main(), Nnet::NumComponents(), and DoBackpropParallelClass::~DoBackpropParallelClass().

                                      {
  KALDI_ASSERT(scale_params.Dim() == this->NumUpdatableComponents());
  int32 i = 0;
  for (int32 j = 0; j < NumComponents(); j++) {
    UpdatableComponent *uc =
        dynamic_cast<UpdatableComponent*>(&(GetComponent(j)));
    const UpdatableComponent *uc_other =
        dynamic_cast<const UpdatableComponent*>(&(other.GetComponent(j)));
    if (uc != NULL) {
      KALDI_ASSERT(uc_other != NULL);
      BaseFloat alpha = scale_params(i);
      uc->Add(alpha, *uc_other);
      i++;
    }
  }
  KALDI_ASSERT(i == scale_params.Dim());
}

AddNnet() [2/3]

void AddNnet	(	BaseFloat	alpha,
		const Nnet &	other
	)

Adds to *this, the other neural net times the scale "alpha".

This applies to UpdatableComponents, and (unlike the other AddNnet function) to SoftmaxComponents.

Definition at line 595 of file nnet-nnet.cc.

References UpdatableComponent::Add(), NonlinearComponent::Add(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, and Nnet::NumComponents().

                                      {
  for (int32 i = 0; i < NumComponents(); i++) {
    UpdatableComponent *uc =
        dynamic_cast<UpdatableComponent*>(&(GetComponent(i)));
    const UpdatableComponent *uc_other =
        dynamic_cast<const UpdatableComponent*>(&(other.GetComponent(i)));
    if (uc != NULL) {
      KALDI_ASSERT(uc_other != NULL);
      uc->Add(alpha, *uc_other);
    }
    NonlinearComponent *nc =
        dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
    const NonlinearComponent *nc_other =
        dynamic_cast<const NonlinearComponent*>(&(other.GetComponent(i)));
    if (nc != NULL) {
      KALDI_ASSERT(nc_other != NULL);
      nc->Add(alpha, *nc_other);
    }
  }
}

AddNnet() [3/3]

void AddNnet	(	BaseFloat	alpha,
		Nnet *	other,
		BaseFloat	beta
	)

This version of AddNnet adds to *this, alpha times *other, and then scales *other by beta.

The reason why we make this a separate function is for multithreading reasons (otherwise you could do AddNnet(alpha, *iter) and then other->Scale(beta).

Definition at line 617 of file nnet-nnet.cc.

References UpdatableComponent::Add(), NonlinearComponent::Add(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, Nnet::NumComponents(), UpdatableComponent::Scale(), and NonlinearComponent::Scale().

                                   {
  for (int32 i = 0; i < NumComponents(); i++) {
    UpdatableComponent *uc =
        dynamic_cast<UpdatableComponent*>(&(GetComponent(i)));
    UpdatableComponent *uc_other =
        dynamic_cast<UpdatableComponent*>(&(other->GetComponent(i)));
    if (uc != NULL) {
      KALDI_ASSERT(uc_other != NULL);
      uc->Add(alpha, *uc_other);
      uc_other->Scale(beta);
    }
    NonlinearComponent *nc =
        dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
    NonlinearComponent *nc_other =
        dynamic_cast<NonlinearComponent*>(&(other->GetComponent(i)));
    if (nc != NULL) {
      KALDI_ASSERT(nc_other != NULL);
      nc->Add(alpha, *nc_other);
      nc_other->Scale(beta);
    }
  }
}

Append()

void Append

(

Component *

new_component

)

Appends this component to the components already in the neural net.

Takes ownership of the pointer.

Definition at line 643 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, and Nnet::SetIndexes().

Referenced by kaldi::nnet2::GiveNnetCorrectTopology(), and Nnet::Nnet().

                                          {
  components_.push_back(new_component);
  SetIndexes();
  Check();
}

Check()

void Check

(

)

const

Definition at line 271 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by Nnet::Append(), Nnet::Collapse(), Nnet::Init(), Nnet::LimitRankOfLastLayer(), kaldi::nnet2::MixupNnet(), Nnet::Nnet(), Nnet::operator=(), Nnet::Read(), Nnet::RemoveDropout(), Nnet::RemovePreconditioning(), Nnet::ResizeOutputLayer(), Nnet::SetComponent(), Nnet::SwitchToOnlinePreconditioning(), kaldi::nnet2::WidenNnet(), Nnet::Write(), and Nnet::~Nnet().

                       {
  for (size_t i = 0; i + 1 < components_.size(); i++) {
    KALDI_ASSERT(components_[i] != NULL);
    int32 output_dim = components_[i]->OutputDim(),
      next_input_dim = components_[i+1]->InputDim();
    KALDI_ASSERT(output_dim == next_input_dim);
    KALDI_ASSERT(components_[i]->Index() == static_cast<int32>(i));
  }
}

Collapse()

void Collapse

(

bool

match_updatableness

)

Where possible, collapse multiple affine or linear components in a sequence into a single one by composing the transforms.

If match_updatableness=true, this will not collapse, say, an AffineComponent with a FixedAffineComponent or FixedLinearComponent. If false, it will collapse them. This function won't necessarily work for all pairs of such layers. It currently only works where one of each pair is an AffineComponent.

Definition at line 730 of file nnet-nnet.cc.

References Nnet::Check(), AffineComponent::CollapseWithNext(), Nnet::components_, rnnlm::i, rnnlm::j, KALDI_LOG, and Nnet::SetIndexes().

Referenced by main(), and Nnet::NumComponents().

                                            {
  int32 num_collapsed = 0;
  bool changed = true;
  while (changed) {
    changed = false;
    for (size_t i = 0; i + 1 < components_.size(); i++) {
      AffineComponent *a1 = dynamic_cast<AffineComponent*>(components_[i]),
          *a2 = dynamic_cast<AffineComponent*>(components_[i + 1]);
      FixedAffineComponent
          *f1 = dynamic_cast<FixedAffineComponent*>(components_[i]),
          *f2 = dynamic_cast<FixedAffineComponent*>(components_[i + 1]);
      Component *c = NULL;
      if (a1 != NULL && a2 != NULL) {
        c = a1->CollapseWithNext(*a2);
      } else if (a1 != NULL && f2 != NULL && !match_updatableness) {
        c = a1->CollapseWithNext(*f2);
      } else if (f1 != NULL && a2 != NULL && !match_updatableness) {
        c = a2->CollapseWithPrevious(*f1);
      }
      if (c != NULL) {
        delete components_[i];
        delete components_[i + 1];
        components_[i] = c;
        // This was causing valgrind errors, so doing it differently.  Either
        // a standard-library bug or I misunderstood something.
        // components_.erase(components_.begin() + i + i,
        //                   components_.begin() + i + 2);
        for (size_t j = i + 1; j + 1 < components_.size(); j++)
          components_[j] = components_[j + 1];
        components_.pop_back();
        changed = true;
        num_collapsed++;
      }
    }
  }
  this->SetIndexes();
  this->Check();
  KALDI_LOG << "Collapsed " << num_collapsed << " components."
            << (num_collapsed == 0 && match_updatableness == true ?
                "  Try --match-updatableness=false." : "");
}

ComponentDotProducts()

void ComponentDotProducts	(	const Nnet &	other,
		VectorBase< BaseFloat > *	dot_prod
	)		const

Definition at line 207 of file nnet-nnet.cc.

References Nnet::components_, VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, and Nnet::NumUpdatableComponents().

Referenced by main(), and Nnet::~Nnet().

                                           {
  KALDI_ASSERT(dot_prod->Dim() == NumUpdatableComponents());
  int32 index = 0;
  for (size_t i = 0; i < components_.size(); i++) {
    UpdatableComponent *uc1 = dynamic_cast<UpdatableComponent*>(components_[i]);
    const UpdatableComponent *uc2 = dynamic_cast<const UpdatableComponent*>(
        &(other.GetComponent(i)));
    KALDI_ASSERT((uc1 != NULL) == (uc2 != NULL));
    if (uc1 != NULL) {
      (*dot_prod)(index) = uc1->DotProduct(*uc2);
      index++;
    }
  }
  KALDI_ASSERT(index == NumUpdatableComponents());
}

ComputeChunkInfo()

void ComputeChunkInfo	(	int32	input_chunk_size,
		int32	num_chunks,
		std::vector< ChunkInfo > *	chunk_info_out
	)		const

Uses the output of the Context() functions of the network, to compute a vector of size NumComponents() + 1 indexed by component-index c, of the chunk-info at the input of each layer c, where the c+1'th element contains the chunk-info at the output of that layer.

The "input_chunk_size" is the time extent of the input. If you want to produce exactly 1 output frame per chunk, then this should equal 1 + LeftContext() + RightContext().

Definition at line 65 of file nnet-nnet.cc.

References Nnet::components_, Component::Context(), Nnet::GetComponent(), rnnlm::i, Nnet::InputDim(), rnnlm::j, KALDI_ASSERT, Nnet::LeftContext(), Nnet::NumComponents(), Nnet::OutputDim(), and Nnet::RightContext().

Referenced by NnetOnlineComputer::Compute(), NnetUpdater::ComputeForMinibatch(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), NnetUpdater::FormatInput(), NnetComputer::NnetComputer(), NnetDiscriminativeUpdater::NnetDiscriminativeUpdater(), and Nnet::NumComponents().

                                                                        {
  // First compute the output-chunk indices for the last component in the
  // network. we assume that the numbering of the input starts from zero.
  int32 output_chunk_size = input_chunk_size - LeftContext() - RightContext();
  KALDI_ASSERT(output_chunk_size > 0);
  std::vector<int32> current_output_inds;
  for (int32 i = 0; i < output_chunk_size; i++)
    current_output_inds.push_back(i + LeftContext());

  (*chunk_info_out).resize(NumComponents() + 1);

  // indexes for last component is empty, since the last component's chunk is
  // always contiguous
  // component's output is always contiguous
  (*chunk_info_out)[NumComponents()] = ChunkInfo(
      GetComponent(NumComponents() - 1).OutputDim(),
      num_chunks, current_output_inds.front(),
      current_output_inds.back());

  std::vector<int32> current_input_inds;
  for (int32 i = NumComponents() - 1; i >= 0; i--) {
    std::vector<int32> current_context = GetComponent(i).Context();
    std::set<int32> current_input_ind_set;
    for (size_t j = 0; j < current_context.size(); j++)
      for (size_t k = 0; k < current_output_inds.size(); k++)
        current_input_ind_set.insert(current_context[j] +
                                     current_output_inds[k]);
    current_output_inds.resize(current_input_ind_set.size());
    std::copy(current_input_ind_set.begin(),
              current_input_ind_set.end(),
              current_output_inds.begin());

    // checking if the vector has contiguous data
    // assign indexes only if the data is not contiguous
    if (current_output_inds.size() !=
        current_output_inds.back() - current_output_inds.front() + 1) {
      (*chunk_info_out)[i] = ChunkInfo(GetComponent(i).InputDim(),
                                       num_chunks,
                                       current_output_inds);
    } else  {
      (*chunk_info_out)[i] = ChunkInfo(GetComponent(i).InputDim(),
                                       num_chunks,
                                       current_output_inds.front(),
                                       current_output_inds.back());
    }
  }

  // TODO: Make a set of components which can deal with data rearrangement.
  // Define this set in an appropriate place so that
  // users adding new components can simply update the list.
  const char *dinit[] = {"SpliceComponent", "SpliceMaxComponent"};
  std::vector< std::string > data_rearrange_components(dinit, dinit + 2);

  // Ensuring that all components until the first component capable of data
  // rearrangement (e.g. SpliceComponent|SpliceMaxComponent) operate on
  // contiguous chunks at the input
  for (size_t i = 0 ; i < NumComponents() ; i++) {
      (*chunk_info_out)[i].MakeOffsetsContiguous();
      // Check if the current component is present in the set of components
      // capable of data rearrangement.
      if (std::find(data_rearrange_components.begin(),
                    data_rearrange_components.end(),
                    components_[i]->Type())
          != data_rearrange_components.end())
          break;
  }

  // sanity testing for chunk_info_out vector
  for (size_t i = 0; i < chunk_info_out->size(); i++) {
    (*chunk_info_out)[i].Check();
    // (*chunk_info_out)[i].ToString();
  }
}

CopyStatsFrom()

void CopyStatsFrom

(

const Nnet &

nnet

)

Copies only the statistics in layers of type NonlinearComponewnt, from this neural net, leaving everything else fixed.

Definition at line 447 of file nnet-nnet.cc.

References NonlinearComponent::Add(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, Nnet::NumComponents(), and NonlinearComponent::Scale().

Referenced by main(), and Nnet::NumComponents().

                                          {
  KALDI_ASSERT(NumComponents() == other.NumComponents());
  for (int32 i = 0; i < NumComponents(); i++) {
    NonlinearComponent *nc =
        dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
    const NonlinearComponent *nc_other =
        dynamic_cast<const NonlinearComponent*>(&(other.GetComponent(i)));
    if (nc != NULL) {
      nc->Scale(0.0);
      nc->Add(1.0, *nc_other);
    }
  }
}

Destroy()

void Destroy

(

)

Definition at line 200 of file nnet-nnet.cc.

References Nnet::components_.

Referenced by Nnet::Init(), Nnet::operator=(), Nnet::Read(), and Nnet::~Nnet().

                   {
  while (!components_.empty()) {
    delete components_.back();
    components_.pop_back();
  }
}

FirstUpdatableComponent()

int32 FirstUpdatableComponent

(

)

const

Returns the index of the lowest-numbered component which is updatable, or NumComponents() if none are updatable.

Definition at line 828 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and Nnet::NumComponents().

Referenced by NnetUpdater::Backprop(), and Nnet::NumComponents().

                                          {
  for (int32 i = 0; i < NumComponents(); i++) {
    if (dynamic_cast<UpdatableComponent*>(components_[i]) != NULL)
      return i;
  }
  return NumComponents();
}

GetComponent() [1/2]

const Component & GetComponent

(

int32

c

)

const

Definition at line 141 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by Nnet::AddNnet(), NnetStats::AddStatsFromNnet(), NnetComputer::Backprop(), NnetDiscriminativeUpdater::Backprop(), NnetUpdater::Backprop(), kaldi::nnet2::CombineNnetsA(), Nnet::ComponentDotProducts(), Nnet::ComputeChunkInfo(), kaldi::nnet2::ComputeObjfAndGradient(), FastNnetCombiner::ComputeObjfAndGradient(), NnetRescaler::ComputeRelevantIndexes(), Nnet::CopyStatsFrom(), kaldi::nnet2::FixNnet(), Nnet::GetLearningRates(), kaldi::nnet2::GetNnetStats(), Nnet::GetParameterDim(), NnetRescaler::GetTargetAvgDeriv(), kaldi::nnet2::GiveNnetCorrectTopology(), kaldi::nnet2::IndexOfSoftmaxLayer(), kaldi::nnet2::InsertComponents(), kaldi::nnet2::LimitRankParallel(), main(), Nnet::NumComponents(), Nnet::NumUpdatableComponents(), LimitRankClass::operator()(), FisherComputationClass::operator()(), NnetComputer::Propagate(), NnetDiscriminativeUpdater::Propagate(), NnetUpdater::Propagate(), NnetOnlineComputer::Propagate(), kaldi::nnet2::ReplaceLastComponents(), NnetRescaler::Rescale(), NnetRescaler::RescaleComponent(), Nnet::ResetGenerators(), Nnet::Scale(), Nnet::ScaleComponents(), Nnet::SetLearningRates(), kaldi::nnet2::SetMaxChange(), Nnet::UnVectorize(), Nnet::Vectorize(), and kaldi::nnet2::WidenNnet().

                                                         {
  KALDI_ASSERT(static_cast<size_t>(component) < components_.size());
  return *(components_[component]);
}

GetComponent() [2/2]

Component & GetComponent

(

int32

c

)

Definition at line 146 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

                                             {
  KALDI_ASSERT(static_cast<size_t>(component) < components_.size());
  return *(components_[component]);
}

GetLearningRates()

void GetLearningRates

(

VectorBase< BaseFloat > *

learning_rates

)

const

Get all the learning rates in the neural net (the output must have dim equal to NumUpdatableComponents()).

Definition at line 476 of file nnet-nnet.cc.

References VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, UpdatableComponent::LearningRate(), Nnet::NumComponents(), and Nnet::NumUpdatableComponents().

Referenced by main(), and Nnet::~Nnet().

                                                                       {
  KALDI_ASSERT(learning_rates->Dim() == this->NumUpdatableComponents());
  int32 i = 0;
  for (int32 j = 0; j < NumComponents(); j++) {
    const UpdatableComponent *uc =
        dynamic_cast<const UpdatableComponent*>(&(GetComponent(j)));
    if (uc!= NULL) {
      (*learning_rates)(i) = uc->LearningRate();
      i++;
    }
  }
  KALDI_ASSERT(i == learning_rates->Dim());
}

GetParameterDim()

int32 GetParameterDim ( ) const

virtual

Definition at line 657 of file nnet-nnet.cc.

References Nnet::GetComponent(), UpdatableComponent::GetParameterDim(), and Nnet::NumComponents().

Referenced by Nnet::Info(), main(), Nnet::UnVectorize(), Nnet::Vectorize(), and Nnet::~Nnet().

                                  {
  int32 ans = 0;
  for (int32 c = 0; c < NumComponents(); c++) {
    const UpdatableComponent *uc = dynamic_cast<const UpdatableComponent*>(
        &(GetComponent(c)));
    if (uc != NULL)
      ans += uc->GetParameterDim();
  }
  return ans;
}

Info()

std::string Info

(

)

const

Definition at line 257 of file nnet-nnet.cc.

References Nnet::components_, Nnet::GetParameterDim(), rnnlm::i, Nnet::InputDim(), Nnet::LeftContext(), Nnet::NumComponents(), Nnet::NumUpdatableComponents(), Nnet::OutputDim(), and Nnet::RightContext().

Referenced by kaldi::nnet2::DoBackprop(), AmNnet::Info(), main(), kaldi::nnet2::UnitTestNnetCompute(), kaldi::nnet2::UnitTestNnetComputeChunked(), and Nnet::~Nnet().

                           {
  std::ostringstream ostr;
  ostr << "num-components " << NumComponents() << std::endl;
  ostr << "num-updatable-components " << NumUpdatableComponents() << std::endl;
  ostr << "left-context " << LeftContext() << std::endl;
  ostr << "right-context " << RightContext() << std::endl;
  ostr << "input-dim " << InputDim() << std::endl;
  ostr << "output-dim " << OutputDim() << std::endl;
  ostr << "parameter-dim " << GetParameterDim() << std::endl;
  for (int32 i = 0; i < NumComponents(); i++)
    ostr << "component " << i << " : " << components_[i]->Info() << std::endl;
  return ostr.str();
}

Init() [1/2]

void Init

(

std::istream &

is

)

Initialize from config file.

Each line of the config is either a comment line starting with whitespace then #, or it is a line that specifies one layer of the network, as accepted by Component::InitFromString(). An example non-comment line is: AffineComponent learning-rate=0.01 l2-penalty=0.001 input-dim=10 output-dim=15 param-stddev=0.1

Definition at line 281 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), KALDI_ASSERT, Component::NewFromString(), and Nnet::SetIndexes().

Referenced by kaldi::nnet2::GenRandomNnet(), AmNnet::Init(), kaldi::nnet2::InsertComponents(), main(), Nnet::Nnet(), and kaldi::nnet2::ReplaceLastComponents().

                              {
  Destroy();
  std::string line;
  /* example config file as follows.  The things in brackets specify the context
     splicing for each layer, and after that is the info about the actual layer.
     Imagine the input dim is 13, and the speaker dim is 40, so (13 x 9) + 40 =  527.
     The config file might be as follows; the lines beginning with # are comments.

     # layer-type layer-options
     AffineLayer 0.01 0.001 527 1000 0.04356
  */
  components_.clear();
  while (getline(is, line)) {
    std::istringstream line_is(line);
    line_is >> std::ws;  // Eat up whitespace.
    if (line_is.peek() == '#' || line_is.eof()) continue;  // Comment or empty.
    Component *c = Component::NewFromString(line);
    KALDI_ASSERT(c != NULL);
    components_.push_back(c);
  }
  SetIndexes();
  Check();
}

Init() [2/2]

void Init

(

std::vector< Component *> *

components

)

This Init method works from a vector of components.

It will take ownership of the pointers and will resize the vector to zero to avoid a chance of the caller deallocating them (but the vector itself is not deleted).

Definition at line 305 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), and Nnet::SetIndexes().

                                                 {
  Destroy();
  components_.swap(*components);
  SetIndexes();
  Check();
}

InputDim()

int32 InputDim

(

)

const

Dimension of the input features, e.g.

13 or 40. Does not take account of frame splicing– that is done with the "chunk" mechanism, where you provide chunks of features over time.

Definition at line 36 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by NnetOnlineComputer::Compute(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), kaldi::nnet2::FormatNnetInput(), Nnet::Info(), Nnet::Nnet(), NnetComputer::NnetComputer(), and Nnet::NumComponents().

                           {
  KALDI_ASSERT(!components_.empty());
  return components_.front()->InputDim();
}

LastUpdatableComponent()

int32 LastUpdatableComponent

(

)

const

Returns the index of the highest-numbered component which is updatable, or -1 if none are updatable.

Definition at line 837 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and Nnet::NumComponents().

Referenced by main(), and Nnet::NumComponents().

                                         {
  for (int32 i = NumComponents() - 1; i >= 0; i--)
    if (dynamic_cast<UpdatableComponent*>(components_[i]) != NULL)
      return i;
  return -1;
}

LeftContext()

int32 LeftContext

(

)

const

Returns the left-context summed over all the Components...

this is the entire left-context in frames, that the network requires.

Definition at line 42 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by NnetOnlineComputer::Compute(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), DecodableAmNnet::DecodableAmNnet(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), NnetUpdater::FormatInput(), kaldi::nnet2::FormatNnetInput(), NnetDiscriminativeUpdater::GetInputFeatures(), Nnet::Info(), main(), kaldi::nnet2::NnetComputationChunked(), NnetComputer::NnetComputer(), Nnet::NumComponents(), kaldi::nnet2::UnitTestNnetCompute(), and kaldi::nnet2::UnitTestNnetComputeChunked().

                              {
  KALDI_ASSERT(!components_.empty());
  int32 ans = 0;
  for (size_t i = 0; i < components_.size(); i++) {
    ans += components_[i]->Context().front();
  }
  return -1*ans;
  // nnet-components return left context as a non-positive integer
  // however the nnet-update, nnet-compute expect a
  // non-negative left context. In addition, the NnetExample also stores data
  // left context as positive integer. To be compatible with these other classes
  // Nnet::LeftContext() returns a non-negative left context.
}

LimitRankOfLastLayer()

void LimitRankOfLastLayer

(

int32

dimension

)

Turns the last affine layer into two layers of the same type, with a smaller dimension in between– we're keeping the top singular values of the matrix.

Definition at line 708 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, KALDI_ERR, AffineComponent::LimitRank(), and Nnet::SetIndexes().

Referenced by Nnet::NumComponents().

                                         {
  for (int32 i = components_.size() - 1; i >= 0; i--) {
    AffineComponent *a = NULL, *b = NULL,
        *c = dynamic_cast<AffineComponent*>(components_[i]);
    if (c != NULL) {
      c->LimitRank(dim, &a, &b);
      delete c;
      components_[i] = a;
      components_.insert(components_.begin() + i + 1, b);
      this->SetIndexes();
      this->Check();
      return;
    }
  }
  KALDI_ERR << "No affine component found in neural net.";
}

NumComponents()

int32 NumComponents ( ) const

inline

Returns number of components– think of this as similar to # of layers, but e.g.

the nonlinearity and the linear part count as separate components, so the number of components will be more than the number of layers.

Definition at line 69 of file nnet-nnet.h.

References Nnet::AddNnet(), Nnet::Collapse(), Nnet::components_, Nnet::ComputeChunkInfo(), Nnet::CopyStatsFrom(), Nnet::FirstUpdatableComponent(), Nnet::GetComponent(), Nnet::InputDim(), Nnet::LastUpdatableComponent(), Nnet::LeftContext(), Nnet::LimitRankOfLastLayer(), Nnet::Nnet(), Nnet::NumUpdatableComponents(), Nnet::OutputDim(), Nnet::RemoveDropout(), Nnet::RemovePreconditioning(), Nnet::Resize(), Nnet::RightContext(), Nnet::Scale(), Nnet::ScaleComponents(), Nnet::SetComponent(), Nnet::SetDropoutScale(), Nnet::SetIndexes(), Nnet::SwitchToOnlinePreconditioning(), and Nnet::ZeroStats().

Referenced by Nnet::AddNnet(), NnetComputer::Backprop(), NnetDiscriminativeUpdater::Backprop(), NnetUpdater::Backprop(), kaldi::nnet2::CombineNnetsA(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), NnetComputer::ComputeLastLayerDeriv(), NnetUpdater::ComputeObjfAndDeriv(), kaldi::nnet2::ComputeObjfAndGradient(), FastNnetCombiner::ComputeObjfAndGradient(), NnetRescaler::ComputeRelevantIndexes(), NnetUpdater::ComputeTotAccuracy(), Nnet::CopyStatsFrom(), Nnet::FirstUpdatableComponent(), kaldi::nnet2::FixNnet(), NnetUpdater::FormatInput(), Nnet::GetLearningRates(), kaldi::nnet2::GetNnetStats(), NnetUpdater::GetOutput(), Nnet::GetParameterDim(), kaldi::nnet2::GiveNnetCorrectTopology(), kaldi::nnet2::IndexOfSoftmaxLayer(), Nnet::Info(), kaldi::nnet2::InsertComponents(), Nnet::LastUpdatableComponent(), NnetDiscriminativeUpdater::LatticeComputations(), kaldi::nnet2::LimitRankParallel(), main(), NnetComputer::NnetComputer(), NnetOnlineComputer::NnetOnlineComputer(), Nnet::NumUpdatableComponents(), FisherComputationClass::operator()(), NnetComputer::Propagate(), NnetDiscriminativeUpdater::Propagate(), NnetOnlineComputer::Propagate(), NnetUpdater::Propagate(), kaldi::nnet2::ReplaceLastComponents(), NnetRescaler::Rescale(), Nnet::ResetGenerators(), Nnet::ResizeOutputLayer(), Nnet::Scale(), Nnet::ScaleComponents(), Nnet::ScaleLearningRates(), Nnet::SetLearningRates(), kaldi::nnet2::SetMaxChange(), Nnet::UnVectorize(), Nnet::Vectorize(), and kaldi::nnet2::WidenNnet().

{ return components_.size(); }

NumUpdatableComponents()

int32 NumUpdatableComponents

(

)

const

Returns the number of updatable components.

Definition at line 413 of file nnet-nnet.cc.

References Nnet::GetComponent(), rnnlm::i, and Nnet::NumComponents().

Referenced by Nnet::AddNnet(), Nnet::ComponentDotProducts(), FisherComputationClass::FisherComputationClass(), Nnet::GetLearningRates(), Nnet::Info(), main(), Nnet::NumComponents(), FisherComputationClass::operator()(), Nnet::ScaleComponents(), Nnet::SetLearningRates(), and kaldi::nnet2::ShrinkNnet().

                                         {
  int32 ans = 0;
  for (int32 i = 0; i < NumComponents(); i++)
    if (dynamic_cast<const UpdatableComponent*>(&(GetComponent(i))) != NULL)
      ans++;
  return ans;
}

operator=()

Nnet & operator=

(

const Nnet &

other

)

Definition at line 247 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), rnnlm::i, and Nnet::SetIndexes().

Referenced by Nnet::Nnet().

                                         {
  Destroy();
  components_.resize(other.components_.size());
  for (size_t i = 0; i < other.components_.size(); i++)
    components_[i] = other.components_[i]->Copy();
  SetIndexes();
  Check();
  return *this;
}

OutputDim()

int32 OutputDim

(

)

const

The output dimension of the network – typically the number of pdfs.

Definition at line 31 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by Nnet::ComputeChunkInfo(), NnetUpdater::ComputeObjfAndDeriv(), Nnet::Info(), AmNnet::Init(), main(), Nnet::Nnet(), Nnet::NumComponents(), and AmNnet::NumPdfs().

                            {
  KALDI_ASSERT(!components_.empty());
  return components_.back()->OutputDim();
}

Read()

void Read	(	std::istream &	is,
		bool	binary
	)

Definition at line 175 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), kaldi::ExpectToken(), kaldi::ReadBasicType(), Component::ReadNew(), and Nnet::SetIndexes().

Referenced by main(), AmNnet::Read(), kaldi::nnet2::UnitTestNnet(), and Nnet::~Nnet().

                                           {
  Destroy();
  ExpectToken(is, binary, "<Nnet>");
  int32 num_components;
  ExpectToken(is, binary, "<NumComponents>");
  ReadBasicType(is, binary, &num_components);
  ExpectToken(is, binary, "<Components>");
  components_.resize(num_components);
  for (int32 c = 0; c < num_components; c++)
    components_[c] = Component::ReadNew(is, binary);
  ExpectToken(is, binary, "</Components>");
  ExpectToken(is, binary, "</Nnet>");
  SetIndexes();
  Check();
}

RemoveDropout()

void RemoveDropout

(

)

Excise any components of type DropoutComponent or AdditiveNoiseComponent.

Definition at line 497 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, KALDI_LOG, and Nnet::SetIndexes().

Referenced by main(), and Nnet::NumComponents().

                         {
  std::vector<Component*> components;
  int32 removed = 0;
  for (size_t i = 0; i < components_.size(); i++) {
    if (dynamic_cast<DropoutComponent*>(components_[i]) != NULL ||
        dynamic_cast<AdditiveNoiseComponent*>(components_[i]) != NULL) {
      delete components_[i];
      removed++;
    } else {
      components.push_back(components_[i]);
    }
  }
  components_ = components;
  if (removed > 0)
    KALDI_LOG << "Removed " << removed << " dropout components.";
  SetIndexes();
  Check();
}

RemovePreconditioning()

void RemovePreconditioning

(

)

Replace any components of type AffineComponentPreconditioned with components of type AffineComponent.

Definition at line 531 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, and Nnet::SetIndexes().

Referenced by main(), and Nnet::NumComponents().

                                 {
  for (size_t i = 0; i < components_.size(); i++) {
    if (dynamic_cast<AffineComponentPreconditioned*>(components_[i]) != NULL) {
      AffineComponent *ac = new AffineComponent(
          *(dynamic_cast<AffineComponent*>(components_[i])));
      delete components_[i];
      components_[i] = ac;
    } else if (dynamic_cast<AffineComponentPreconditionedOnline*>(
        components_[i]) != NULL) {
      AffineComponent *ac = new AffineComponent(
          *(dynamic_cast<AffineComponent*>(components_[i])));
      delete components_[i];
      components_[i] = ac;
    }
  }
  SetIndexes();
  Check();
}

ResetGenerators()

void ResetGenerators

(

)

Definition at line 683 of file nnet-nnet.cc.

References Nnet::GetComponent(), Nnet::NumComponents(), and RandomComponent::ResetGenerator().

Referenced by Nnet::~Nnet().

                           {
  // resets random-number generators for all random
  // components.
  for (int32 c = 0; c < NumComponents(); c++) {
    RandomComponent *rc = dynamic_cast<RandomComponent*>(
        &(GetComponent(c)));
    if (rc != NULL)
      rc->ResetGenerator();
  }
}

Resize()

void Resize

(

int32

num_components

)

Removes final components from the neural network (used for debugging).

Definition at line 490 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by main(), and Nnet::NumComponents().

                                {
  KALDI_ASSERT(new_size <= static_cast<int32>(components_.size()));
  for (size_t i = new_size; i < components_.size(); i++)
    delete components_[i];
  components_.resize(new_size);
}

ResizeOutputLayer()

void ResizeOutputLayer

(

int32

new_num_pdfs

)

This function is used when doing transfer learning to a new system.

It resizes the final affine and softmax components. If your system has a SumGroupComponent before the final softmax, it will be discarded.

Definition at line 356 of file nnet-nnet.cc.

References Nnet::Check(), AffineComponent::CollapseWithNext(), Nnet::components_, AffineComponent::InputDim(), KALDI_ASSERT, KALDI_ERR, Nnet::NumComponents(), AffineComponent::Resize(), and Nnet::SetIndexes().

Referenced by AmNnet::ResizeOutputLayer(), and Nnet::~Nnet().

                                               {
  KALDI_ASSERT(new_num_pdfs > 0);
  KALDI_ASSERT(NumComponents() > 2);
  int32 nc = NumComponents();
  SumGroupComponent *sgc =
      dynamic_cast<SumGroupComponent*>(components_[nc - 1]);
  if (sgc != NULL) {
    // Remove it.  We'll resize things later.
    delete sgc;
    components_.erase(components_.begin() + nc - 1,
                      components_.begin() + nc);
    nc--;
  }
  SoftmaxComponent *sc;
  if ((sc = dynamic_cast<SoftmaxComponent*>(components_[nc - 1])) == NULL)
    KALDI_ERR << "Expected last component to be SoftmaxComponent.";

  // check if nc-1 has a FixedScaleComponent
  bool has_fixed_scale_component = false;
  int32 fixed_scale_component_index = -1;
  int32 final_affine_component_index = nc - 2;
  int32 softmax_component_index = nc - 1;
  FixedScaleComponent *fsc =
      dynamic_cast<FixedScaleComponent*>(
          components_[final_affine_component_index]);
  if (fsc != NULL)  {
    has_fixed_scale_component = true;
    fixed_scale_component_index = nc - 2;
    final_affine_component_index = nc - 3;
  }
  // note: it could be child class of AffineComponent.
  AffineComponent *ac = dynamic_cast<AffineComponent*>(
      components_[final_affine_component_index]);
  if (ac == NULL)
    KALDI_ERR << "Network doesn't have expected structure (didn't find final "
              << "AffineComponent).";
  if (has_fixed_scale_component)  {
    // collapse the fixed_scale_component with the affine_component before it
    AffineComponent *ac_new =
        dynamic_cast<AffineComponent*>(ac->CollapseWithNext(*fsc));
    KALDI_ASSERT(ac_new != NULL);
    delete fsc;
    delete ac;
    components_.erase(components_.begin() + fixed_scale_component_index,
                      components_.begin() + (fixed_scale_component_index + 1));
    components_[final_affine_component_index] = ac_new;
    ac = ac_new;
    softmax_component_index = softmax_component_index - 1;
  }
  ac->Resize(ac->InputDim(), new_num_pdfs);
  // Remove the softmax component, and replace it with a new one
  delete components_[softmax_component_index];
  components_[softmax_component_index] = new SoftmaxComponent(new_num_pdfs);
  this->SetIndexes();  // used for debugging
  this->Check();
}

RightContext()

int32 RightContext

(

)

const

Returns the right-context summed over all the Components...

this is the entire right-context in frames, that the network requires.

Definition at line 56 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by NnetOnlineComputer::Compute(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), DecodableAmNnet::DecodableAmNnet(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), NnetUpdater::FormatInput(), kaldi::nnet2::FormatNnetInput(), NnetDiscriminativeUpdater::GetInputFeatures(), Nnet::Info(), main(), kaldi::nnet2::NnetComputationChunked(), NnetComputer::NnetComputer(), Nnet::NumComponents(), kaldi::nnet2::UnitTestNnetCompute(), and kaldi::nnet2::UnitTestNnetComputeChunked().

                               {
  KALDI_ASSERT(!components_.empty());
  int32 ans = 0;
  for (size_t i = 0; i < components_.size(); i++) {
    ans += components_[i]->Context().back();
  }
  return ans;
}

Scale()

void Scale

(

BaseFloat

scale

)

Scales all the Components with the same scale.

This applies to UpdatableComponents, and (unlike the ScaleComponents function) to SoftmaxComponents.

Definition at line 436 of file nnet-nnet.cc.

References Nnet::GetComponent(), rnnlm::i, Nnet::NumComponents(), UpdatableComponent::Scale(), and NonlinearComponent::Scale().

Referenced by main(), and Nnet::NumComponents().

                                {
  for (int32 i = 0; i < NumComponents(); i++) {
    UpdatableComponent *uc =
        dynamic_cast<UpdatableComponent*>(&(GetComponent(i)));
    if (uc != NULL) uc->Scale(scale);
    NonlinearComponent *nc =
        dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
    if (nc != NULL) nc->Scale(scale);
  }
}

ScaleComponents()

void ScaleComponents

(

const VectorBase< BaseFloat > &

scales

)

Scales the parameters of each of the updatable components.

Here, scale_params is a vector of size equal to NumUpdatableComponents()

Definition at line 421 of file nnet-nnet.cc.

References VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), Nnet::NumUpdatableComponents(), and UpdatableComponent::Scale().

Referenced by kaldi::nnet2::CombineNnets(), FastNnetCombiner::CombineNnets(), kaldi::nnet2::ComputeObjfAndGradient(), kaldi::nnet2::GetUpdateDirection(), main(), Nnet::NumComponents(), and kaldi::nnet2::ShrinkNnet().

                                                                    {
  KALDI_ASSERT(scale_params.Dim() == this->NumUpdatableComponents());
  int32 i = 0;
  for (int32 j = 0; j < NumComponents(); j++) {
    UpdatableComponent *uc =
        dynamic_cast<UpdatableComponent*>(&(GetComponent(j)));
    if (uc!= NULL) {
      uc->Scale(scale_params(i));
      i++;
    }
  }
  KALDI_ASSERT(i == scale_params.Dim());
}

ScaleLearningRates() [1/2]

void ScaleLearningRates

(

BaseFloat

factor

)

Scale all the learning rates in the neural net by this factor.

Definition at line 313 of file nnet-nnet.cc.

References Nnet::components_, KALDI_LOG, UpdatableComponent::LearningRate(), Nnet::NumComponents(), and UpdatableComponent::SetLearningRate().

Referenced by main(), and Nnet::~Nnet().

                                              {
  std::ostringstream ostr;
  for (int32 c = 0; c < NumComponents(); c++) {
    UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[c]);
    if (uc != NULL) {  // Updatable component...
      uc->SetLearningRate(uc->LearningRate() * factor);
      ostr << uc->LearningRate() << " ";
    }
  }
  KALDI_LOG << "Scaled learning rates by " << factor
            << ", new learning rates are "
            << ostr.str();
}

ScaleLearningRates() [2/2]

void ScaleLearningRates

(

std::map< std::string, BaseFloat >

scale_factors

)

Scale all the learning rates in the neural net by the factors indexed by the type of component.

Definition at line 327 of file nnet-nnet.cc.

References Nnet::components_, KALDI_LOG, UpdatableComponent::LearningRate(), Nnet::NumComponents(), UpdatableComponent::SetLearningRate(), and Component::Type().

                                                                        {
  std::ostringstream ostr;
  for (int32 c = 0; c < NumComponents(); c++) {
    UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[c]);
    if (uc != NULL) {  // Updatable component...
      // check if scaling factor was specified for a component of this type
      std::map<std::string, BaseFloat>::const_iterator lr_iterator =
        scale_factors.find(uc->Type());
      if (lr_iterator != scale_factors.end())  {
        uc->SetLearningRate(uc->LearningRate() * lr_iterator->second);
        ostr << uc->LearningRate() << " ";
      }
    }
  }
  KALDI_LOG << "Scaled learning rates by component-type specific factor, "
            << "new learning rates are "
            << ostr.str();
}

SetComponent()

void SetComponent	(	int32	c,
		Component *	component
	)

Sets the c'th component to "component", taking ownership of the pointer and deleting the corresponding one that we own.

Definition at line 649 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, KALDI_ASSERT, and Nnet::SetIndexes().

Referenced by Nnet::NumComponents().

                                                     {
  KALDI_ASSERT(static_cast<size_t>(c) < components_.size());
  delete components_[c];
  components_[c] = component;
  SetIndexes();
  Check();  // Check that all the dimensions still match up.
}

SetDropoutScale()

void SetDropoutScale

(

BaseFloat

scale

)

Calls SetDropoutScale for all the dropout nodes.

Definition at line 516 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, KALDI_LOG, and DropoutComponent::SetDropoutScale().

Referenced by main(), and Nnet::NumComponents().

                                          {
  size_t n_set = 0;
  for (size_t i = 0; i < components_.size(); i++) {
    DropoutComponent *dc =
        dynamic_cast<DropoutComponent*>(components_[i]);
    if (dc != NULL) {
      dc->SetDropoutScale(scale);
      n_set++;
    }
  }
  KALDI_LOG << "Set dropout scale to " << scale
            << " for " << n_set << " components.";
}

SetIndexes()

void SetIndexes

(

)

Sets the index_ values of the components.

Definition at line 725 of file nnet-nnet.cc.

References Nnet::components_, and rnnlm::i.

Referenced by Nnet::Append(), Nnet::Collapse(), Nnet::Init(), Nnet::LimitRankOfLastLayer(), Nnet::Nnet(), Nnet::NumComponents(), Nnet::operator=(), Nnet::Read(), Nnet::RemoveDropout(), Nnet::RemovePreconditioning(), Nnet::ResizeOutputLayer(), Nnet::SetComponent(), and Nnet::SwitchToOnlinePreconditioning().

                      {
  for (size_t i = 0; i < components_.size(); i++)
    components_[i]->SetIndex(i);
}

SetLearningRates() [1/2]

void SetLearningRates

(

BaseFloat

learning_rates

)

Set all the learning rates in the neural net to this value.

Definition at line 346 of file nnet-nnet.cc.

References Nnet::components_, KALDI_LOG, Nnet::NumComponents(), and UpdatableComponent::SetLearningRate().

Referenced by main(), and Nnet::~Nnet().

                                                   {
  for (int32 c = 0; c < NumComponents(); c++) {
    UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[c]);
    if (uc != NULL) {  // Updatable component...
      uc->SetLearningRate(learning_rate);
    }
  }
  KALDI_LOG << "Set learning rates to " << learning_rate;
}

SetLearningRates() [2/2]

void SetLearningRates

(

const VectorBase< BaseFloat > &

learning_rates

)

Set all the learning rates in the neural net to these values (one for each updatable layer).

Definition at line 461 of file nnet-nnet.cc.

References VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, VectorBase< Real >::Min(), Nnet::NumComponents(), Nnet::NumUpdatableComponents(), and UpdatableComponent::SetLearningRate().

                                                                       {
  KALDI_ASSERT(learning_rates.Dim() == this->NumUpdatableComponents());
  KALDI_ASSERT(learning_rates.Min() >= 0.0);  // we allow zero learning rate.
  int32 i = 0;
  for (int32 j = 0; j < NumComponents(); j++) {
    UpdatableComponent *uc =
        dynamic_cast<UpdatableComponent*>(&(GetComponent(j)));
    if (uc!= NULL) {
      uc->SetLearningRate(learning_rates(i));
      i++;
    }
  }
  KALDI_ASSERT(i == learning_rates.Dim());
}

SetZero()

void SetZero

(

bool

treat_as_gradient

)

Definition at line 151 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, NonlinearComponent::Scale(), and UpdatableComponent::SetZero().

Referenced by kaldi::nnet2::ComputeNnetGradient(), kaldi::nnet2::ComputeObjfAndGradient(), FastNnetCombiner::ComputeObjfAndGradient(), DoBackpropParallelClass::DoBackpropParallelClass(), main(), FisherComputationClass::operator()(), and Nnet::~Nnet().

                                         {
  for (size_t i = 0; i < components_.size(); i++) {
    UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[i]);
    if (uc != NULL) uc->SetZero(treat_as_gradient);
    NonlinearComponent *nc = dynamic_cast<NonlinearComponent*>(components_[i]);
    if (nc != NULL) nc->Scale(0.0);
  }
}

SwitchToOnlinePreconditioning()

void SwitchToOnlinePreconditioning	(	int32	rank_in,
		int32	rank_out,
		int32	update_period,
		BaseFloat	num_samples_history,
		BaseFloat	alpha
	)

Replaces any components of type AffineComponent or derived classes, with components of type AffineComponentPreconditionedOnline.

E.g. rank_in = 20, rank_out = 80, num_samples_history = 2000.0, alpha = 4.0

Definition at line 551 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, KALDI_LOG, and Nnet::SetIndexes().

Referenced by main(), and Nnet::NumComponents().

                                                          {
  int32 switched = 0;
  for (size_t i = 0; i < components_.size(); i++) {
    if (dynamic_cast<AffineComponent*>(components_[i]) != NULL) {
      AffineComponentPreconditionedOnline *ac =
          new AffineComponentPreconditionedOnline(
              *(dynamic_cast<AffineComponent*>(components_[i])),
              rank_in, rank_out, update_period, num_samples_history, alpha);
      delete components_[i];
      components_[i] = ac;
      switched++;
    }
  }
  KALDI_LOG << "Switched " << switched << " components to use online "
            << "preconditioning, with (input, output) rank = "
            << rank_in << ", " << rank_out << " and num_samples_history = "
            << num_samples_history;
  SetIndexes();
  Check();
}

UnVectorize()

void UnVectorize ( const VectorBase< BaseFloat > &  params )

virtual

Definition at line 694 of file nnet-nnet.cc.

References Nnet::GetComponent(), Nnet::GetParameterDim(), UpdatableComponent::GetParameterDim(), KALDI_ASSERT, Nnet::NumComponents(), VectorBase< Real >::Range(), and UpdatableComponent::UnVectorize().

Referenced by Nnet::~Nnet().

                                                          {
  int32 offset = 0;
  for (int32 c = 0; c < NumComponents(); c++) {
    UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(
        &(GetComponent(c)));
    if (uc != NULL) {
      int32 size = uc->GetParameterDim();
      uc->UnVectorize(params.Range(offset, size));
      offset += size;
    }
  }
  KALDI_ASSERT(offset == GetParameterDim());
}

Vectorize()

void Vectorize ( VectorBase< BaseFloat > *  params ) const

virtual

Definition at line 668 of file nnet-nnet.cc.

References Nnet::GetComponent(), Nnet::GetParameterDim(), UpdatableComponent::GetParameterDim(), KALDI_ASSERT, Nnet::NumComponents(), and UpdatableComponent::Vectorize().

Referenced by Nnet::~Nnet().

                                                        {
  int32 offset = 0;
  for (int32 c = 0; c < NumComponents(); c++) {
    const UpdatableComponent *uc = dynamic_cast<const UpdatableComponent*>(
        &(GetComponent(c)));
    if (uc != NULL) {
      int32 size = uc->GetParameterDim();
      SubVector<BaseFloat> temp(*params, offset, size);
      uc->Vectorize(&temp);
      offset += size;
    }
  }
  KALDI_ASSERT(offset == GetParameterDim());
}

Write()

void Write	(	std::ostream &	os,
		bool	binary
	)		const

Definition at line 160 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, kaldi::WriteBasicType(), and kaldi::WriteToken().

Referenced by kaldi::nnet2::UnitTestNnet(), AmNnet::Write(), and Nnet::~Nnet().

                                                  {
  Check();
  WriteToken(os, binary, "<Nnet>");
  int32 num_components = components_.size();
  WriteToken(os, binary, "<NumComponents>");
  WriteBasicType(os, binary, num_components);
  WriteToken(os, binary, "<Components>");
  for (int32 c = 0; c < num_components; c++) {
    components_[c]->Write(os, binary);
    if (!binary) os << std::endl;
  }
  WriteToken(os, binary, "</Components>");
  WriteToken(os, binary, "</Nnet>");
}

ZeroStats()

void ZeroStats

(

)

Definition at line 192 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and NonlinearComponent::Scale().

Referenced by main(), and Nnet::NumComponents().

                     {
  for (size_t i = 0; i < components_.size(); i++) {
    NonlinearComponent *nonlinear_component =
        dynamic_cast<NonlinearComponent*>(components_[i]);
    if (nonlinear_component != NULL)
      nonlinear_component->Scale(0.0);  // Zero the stats this way.
  }
}

Friends And Related Function Documentation

DecodableNnet

friend class DecodableNnet

friend

Definition at line 288 of file nnet-nnet.h.

NnetUpdater

friend class NnetUpdater

friend

Definition at line 287 of file nnet-nnet.h.

Member Data Documentation

components_

std::vector<Component*> components_

private

Definition at line 290 of file nnet-nnet.h.

Referenced by Nnet::Append(), Nnet::Check(), Nnet::Collapse(), Nnet::ComponentDotProducts(), Nnet::ComputeChunkInfo(), Nnet::Destroy(), Nnet::FirstUpdatableComponent(), Nnet::GetComponent(), Nnet::Info(), Nnet::Init(), Nnet::InputDim(), Nnet::LastUpdatableComponent(), Nnet::LeftContext(), Nnet::LimitRankOfLastLayer(), Nnet::Nnet(), Nnet::NumComponents(), Nnet::operator=(), Nnet::OutputDim(), Nnet::Read(), Nnet::RemoveDropout(), Nnet::RemovePreconditioning(), Nnet::Resize(), Nnet::ResizeOutputLayer(), Nnet::RightContext(), Nnet::ScaleLearningRates(), Nnet::SetComponent(), Nnet::SetDropoutScale(), Nnet::SetIndexes(), Nnet::SetLearningRates(), Nnet::SetZero(), Nnet::SwitchToOnlinePreconditioning(), Nnet::Write(), and Nnet::ZeroStats().

---

The documentation for this class was generated from the following files:

• nnet2/nnet-nnet.h
• nnet2/nnet-nnet.cc