All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
Nnet Class Reference

#include <nnet-nnet.h>

Collaboration diagram for Nnet:

Public Member Functions

int32 NumComponents () const
 Returns number of components– think of this as similar to # of layers, but e.g. More...
 
const ComponentGetComponent (int32 c) const
 
ComponentGetComponent (int32 c)
 
void SetComponent (int32 c, Component *component)
 Sets the c'th component to "component", taking ownership of the pointer and deleting the corresponding one that we own. More...
 
int32 LeftContext () const
 Returns the left-context summed over all the Components... More...
 
int32 RightContext () const
 Returns the right-context summed over all the Components... More...
 
int32 OutputDim () const
 The output dimension of the network – typically the number of pdfs. More...
 
int32 InputDim () const
 Dimension of the input features, e.g. More...
 
void ComputeChunkInfo (int32 input_chunk_size, int32 num_chunks, std::vector< ChunkInfo > *chunk_info_out) const
 Uses the output of the Context() functions of the network, to compute a vector of size NumComponents() + 1 indexed by component-index c, of the chunk-info at the input of each layer c, where the c+1'th element contains the chunk-info at the output of that layer. More...
 
void ZeroStats ()
 
void CopyStatsFrom (const Nnet &nnet)
 Copies only the statistics in layers of type NonlinearComponewnt, from this neural net, leaving everything else fixed. More...
 
int32 FirstUpdatableComponent () const
 Returns the index of the lowest-numbered component which is updatable, or NumComponents() if none are updatable. More...
 
int32 LastUpdatableComponent () const
 Returns the index of the highest-numbered component which is updatable, or -1 if none are updatable. More...
 
int32 NumUpdatableComponents () const
 Returns the number of updatable components. More...
 
void ScaleComponents (const VectorBase< BaseFloat > &scales)
 Scales the parameters of each of the updatable components. More...
 
void RemoveDropout ()
 Excise any components of type DropoutComponent or AdditiveNoiseComponent. More...
 
void SetDropoutScale (BaseFloat scale)
 Calls SetDropoutScale for all the dropout nodes. More...
 
void RemovePreconditioning ()
 Replace any components of type AffineComponentPreconditioned with components of type AffineComponent. More...
 
void SwitchToOnlinePreconditioning (int32 rank_in, int32 rank_out, int32 update_period, BaseFloat num_samples_history, BaseFloat alpha)
 Replaces any components of type AffineComponent or derived classes, with components of type AffineComponentPreconditionedOnline. More...
 
void AddNnet (const VectorBase< BaseFloat > &scales, const Nnet &other)
 For each updatatable component, adds to it the corresponding element of "other" times the appropriate element of "scales" (which has the same format as for ScaleComponents(), i.e. More...
 
void Scale (BaseFloat scale)
 Scales all the Components with the same scale. More...
 
void AddNnet (BaseFloat alpha, const Nnet &other)
 Adds to *this, the other neural net times the scale "alpha". More...
 
void LimitRankOfLastLayer (int32 dimension)
 Turns the last affine layer into two layers of the same type, with a smaller dimension in between– we're keeping the top singular values of the matrix. More...
 
void AddNnet (BaseFloat alpha, Nnet *other, BaseFloat beta)
 This version of AddNnet adds to *this, alpha times *other, and then scales *other by beta. More...
 
void Resize (int32 num_components)
 Removes final components from the neural network (used for debugging). More...
 
void Collapse (bool match_updatableness)
 Where possible, collapse multiple affine or linear components in a sequence into a single one by composing the transforms. More...
 
void SetIndexes ()
 Sets the index_ values of the components. More...
 
 Nnet (const Nnet &other)
 
 Nnet (const Nnet &nnet1, const Nnet &nnet2)
 
 Nnet ()
 
Nnetoperator= (const Nnet &other)
 
void Init (std::istream &is)
 Initialize from config file. More...
 
void Init (std::vector< Component * > *components)
 This Init method works from a vector of components. More...
 
void Append (Component *new_component)
 Appends this component to the components already in the neural net. More...
 
virtual ~Nnet ()
 
std::string Info () const
 
void Destroy ()
 
void Write (std::ostream &os, bool binary) const
 
void Read (std::istream &is, bool binary)
 
void SetZero (bool treat_as_gradient)
 
void ResizeOutputLayer (int32 new_num_pdfs)
 This function is used when doing transfer learning to a new system. More...
 
void ScaleLearningRates (BaseFloat factor)
 Scale all the learning rates in the neural net by this factor. More...
 
void ScaleLearningRates (std::map< std::string, BaseFloat > scale_factors)
 Scale all the learning rates in the neural net by the factors indexed by the type of component. More...
 
void SetLearningRates (BaseFloat learning_rates)
 Set all the learning rates in the neural net to this value. More...
 
void SetLearningRates (const VectorBase< BaseFloat > &learning_rates)
 Set all the learning rates in the neural net to these values (one for each updatable layer). More...
 
void GetLearningRates (VectorBase< BaseFloat > *learning_rates) const
 Get all the learning rates in the neural net (the output must have dim equal to NumUpdatableComponents()). More...
 
void ComponentDotProducts (const Nnet &other, VectorBase< BaseFloat > *dot_prod) const
 
void Check () const
 
void ResetGenerators ()
 
virtual int32 GetParameterDim () const
 
virtual void Vectorize (VectorBase< BaseFloat > *params) const
 
virtual void UnVectorize (const VectorBase< BaseFloat > &params)
 

Private Attributes

std::vector< Component * > components_
 

Friends

class NnetUpdater
 
class DecodableNnet
 

Detailed Description

Definition at line 63 of file nnet-nnet.h.

Constructor & Destructor Documentation

Nnet ( const Nnet other)

Definition at line 226 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, and Nnet::SetIndexes().

226  : components_(other.components_.size()) {
227  for (size_t i = 0; i < other.components_.size(); i++)
228  components_[i] = other.components_[i]->Copy();
229  SetIndexes();
230  Check();
231 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
Nnet ( const Nnet nnet1,
const Nnet nnet2 
)

Definition at line 233 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, Nnet::InputDim(), KALDI_ERR, Nnet::OutputDim(), and Nnet::SetIndexes().

233  {
234  int32 dim1 = other1.OutputDim(), dim2 = other2.InputDim();
235  if (dim1 != dim2)
236  KALDI_ERR << "Concatenating neural nets: dimension mismatch "
237  << dim1 << " vs. " << dim2;
238  for (size_t i = 0; i < other1.components_.size(); i++)
239  components_.push_back(other1.components_[i]->Copy());
240  for (size_t i = 0; i < other2.components_.size(); i++)
241  components_.push_back(other2.components_[i]->Copy());
242  SetIndexes();
243  Check();
244 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ERR
Definition: kaldi-error.h:127
void Check() const
Definition: nnet-nnet.cc:271
Nnet ( )
inline

Definition at line 203 of file nnet-nnet.h.

203 {}
virtual ~Nnet ( )
inlinevirtual

Definition at line 225 of file nnet-nnet.h.

References Nnet::Destroy().

225 { Destroy(); }

Member Function Documentation

void AddNnet ( const VectorBase< BaseFloat > &  scales,
const Nnet other 
)

For each updatatable component, adds to it the corresponding element of "other" times the appropriate element of "scales" (which has the same format as for ScaleComponents(), i.e.

one entry for each updatable component).

Definition at line 576 of file nnet-nnet.cc.

References UpdatableComponent::Add(), VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), and Nnet::NumUpdatableComponents().

Referenced by kaldi::nnet2::AddDirection(), kaldi::nnet2::CombineNnets(), FastNnetCombiner::CombineNnets(), kaldi::nnet2::GetUpdateDirection(), main(), DiscTrainParallelClass::~DiscTrainParallelClass(), and DoBackpropParallelClass::~DoBackpropParallelClass().

577  {
578  KALDI_ASSERT(scale_params.Dim() == this->NumUpdatableComponents());
579  int32 i = 0;
580  for (int32 j = 0; j < NumComponents(); j++) {
581  UpdatableComponent *uc =
582  dynamic_cast<UpdatableComponent*>(&(GetComponent(j)));
583  const UpdatableComponent *uc_other =
584  dynamic_cast<const UpdatableComponent*>(&(other.GetComponent(j)));
585  if (uc != NULL) {
586  KALDI_ASSERT(uc_other != NULL);
587  BaseFloat alpha = scale_params(i);
588  uc->Add(alpha, *uc_other);
589  i++;
590  }
591  }
592  KALDI_ASSERT(i == scale_params.Dim());
593 }
float BaseFloat
Definition: kaldi-types.h:29
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents() const
Returns the number of updatable components.
Definition: nnet-nnet.cc:413
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void AddNnet ( BaseFloat  alpha,
const Nnet other 
)

Adds to *this, the other neural net times the scale "alpha".

This applies to UpdatableComponents, and (unlike the other AddNnet function) to SoftmaxComponents.

Definition at line 595 of file nnet-nnet.cc.

References UpdatableComponent::Add(), NonlinearComponent::Add(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, and Nnet::NumComponents().

596  {
597  for (int32 i = 0; i < NumComponents(); i++) {
598  UpdatableComponent *uc =
599  dynamic_cast<UpdatableComponent*>(&(GetComponent(i)));
600  const UpdatableComponent *uc_other =
601  dynamic_cast<const UpdatableComponent*>(&(other.GetComponent(i)));
602  if (uc != NULL) {
603  KALDI_ASSERT(uc_other != NULL);
604  uc->Add(alpha, *uc_other);
605  }
606  NonlinearComponent *nc =
607  dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
608  const NonlinearComponent *nc_other =
609  dynamic_cast<const NonlinearComponent*>(&(other.GetComponent(i)));
610  if (nc != NULL) {
611  KALDI_ASSERT(nc_other != NULL);
612  nc->Add(alpha, *nc_other);
613  }
614  }
615 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void AddNnet ( BaseFloat  alpha,
Nnet other,
BaseFloat  beta 
)

This version of AddNnet adds to *this, alpha times *other, and then scales *other by beta.

The reason why we make this a separate function is for multithreading reasons (otherwise you could do AddNnet(alpha, *iter) and then other->Scale(beta).

Definition at line 617 of file nnet-nnet.cc.

References UpdatableComponent::Add(), NonlinearComponent::Add(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, Nnet::NumComponents(), UpdatableComponent::Scale(), and NonlinearComponent::Scale().

619  {
620  for (int32 i = 0; i < NumComponents(); i++) {
621  UpdatableComponent *uc =
622  dynamic_cast<UpdatableComponent*>(&(GetComponent(i)));
623  UpdatableComponent *uc_other =
624  dynamic_cast<UpdatableComponent*>(&(other->GetComponent(i)));
625  if (uc != NULL) {
626  KALDI_ASSERT(uc_other != NULL);
627  uc->Add(alpha, *uc_other);
628  uc_other->Scale(beta);
629  }
630  NonlinearComponent *nc =
631  dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
632  NonlinearComponent *nc_other =
633  dynamic_cast<NonlinearComponent*>(&(other->GetComponent(i)));
634  if (nc != NULL) {
635  KALDI_ASSERT(nc_other != NULL);
636  nc->Add(alpha, *nc_other);
637  nc_other->Scale(beta);
638  }
639  }
640 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void Append ( Component new_component)

Appends this component to the components already in the neural net.

Takes ownership of the pointer.

Definition at line 643 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, and Nnet::SetIndexes().

Referenced by kaldi::nnet2::GiveNnetCorrectTopology().

643  {
644  components_.push_back(new_component);
645  SetIndexes();
646  Check();
647 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
void Check ( ) const

Definition at line 271 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by Nnet::Append(), Nnet::Collapse(), Nnet::Init(), Nnet::LimitRankOfLastLayer(), kaldi::nnet2::MixupNnet(), Nnet::Nnet(), Nnet::operator=(), Nnet::Read(), Nnet::RemoveDropout(), Nnet::RemovePreconditioning(), Nnet::ResizeOutputLayer(), Nnet::SetComponent(), Nnet::SwitchToOnlinePreconditioning(), kaldi::nnet2::WidenNnet(), and Nnet::Write().

271  {
272  for (size_t i = 0; i + 1 < components_.size(); i++) {
273  KALDI_ASSERT(components_[i] != NULL);
274  int32 output_dim = components_[i]->OutputDim(),
275  next_input_dim = components_[i+1]->InputDim();
276  KALDI_ASSERT(output_dim == next_input_dim);
277  KALDI_ASSERT(components_[i]->Index() == static_cast<int32>(i));
278  }
279 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Collapse ( bool  match_updatableness)

Where possible, collapse multiple affine or linear components in a sequence into a single one by composing the transforms.

If match_updatableness=true, this will not collapse, say, an AffineComponent with a FixedAffineComponent or FixedLinearComponent. If false, it will collapse them. This function won't necessarily work for all pairs of such layers. It currently only works where one of each pair is an AffineComponent.

Definition at line 730 of file nnet-nnet.cc.

References Nnet::Check(), AffineComponent::CollapseWithNext(), Nnet::components_, rnnlm::i, rnnlm::j, KALDI_LOG, and Nnet::SetIndexes().

Referenced by main().

730  {
731  int32 num_collapsed = 0;
732  bool changed = true;
733  while (changed) {
734  changed = false;
735  for (size_t i = 0; i + 1 < components_.size(); i++) {
736  AffineComponent *a1 = dynamic_cast<AffineComponent*>(components_[i]),
737  *a2 = dynamic_cast<AffineComponent*>(components_[i + 1]);
738  FixedAffineComponent
739  *f1 = dynamic_cast<FixedAffineComponent*>(components_[i]),
740  *f2 = dynamic_cast<FixedAffineComponent*>(components_[i + 1]);
741  Component *c = NULL;
742  if (a1 != NULL && a2 != NULL) {
743  c = a1->CollapseWithNext(*a2);
744  } else if (a1 != NULL && f2 != NULL && !match_updatableness) {
745  c = a1->CollapseWithNext(*f2);
746  } else if (f1 != NULL && a2 != NULL && !match_updatableness) {
747  c = a2->CollapseWithPrevious(*f1);
748  }
749  if (c != NULL) {
750  delete components_[i];
751  delete components_[i + 1];
752  components_[i] = c;
753  // This was causing valgrind errors, so doing it differently. Either
754  // a standard-library bug or I misunderstood something.
755  // components_.erase(components_.begin() + i + i,
756  // components_.begin() + i + 2);
757  for (size_t j = i + 1; j + 1 < components_.size(); j++)
758  components_[j] = components_[j + 1];
759  components_.pop_back();
760  changed = true;
761  num_collapsed++;
762  }
763  }
764  }
765  this->SetIndexes();
766  this->Check();
767  KALDI_LOG << "Collapsed " << num_collapsed << " components."
768  << (num_collapsed == 0 && match_updatableness == true ?
769  " Try --match-updatableness=false." : "");
770 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
#define KALDI_LOG
Definition: kaldi-error.h:133
void ComponentDotProducts ( const Nnet other,
VectorBase< BaseFloat > *  dot_prod 
) const

Definition at line 207 of file nnet-nnet.cc.

References Nnet::components_, VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, and Nnet::NumUpdatableComponents().

Referenced by main().

209  {
210  KALDI_ASSERT(dot_prod->Dim() == NumUpdatableComponents());
211  int32 index = 0;
212  for (size_t i = 0; i < components_.size(); i++) {
213  UpdatableComponent *uc1 = dynamic_cast<UpdatableComponent*>(components_[i]);
214  const UpdatableComponent *uc2 = dynamic_cast<const UpdatableComponent*>(
215  &(other.GetComponent(i)));
216  KALDI_ASSERT((uc1 != NULL) == (uc2 != NULL));
217  if (uc1 != NULL) {
218  (*dot_prod)(index) = uc1->DotProduct(*uc2);
219  index++;
220  }
221  }
223 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents() const
Returns the number of updatable components.
Definition: nnet-nnet.cc:413
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
void ComputeChunkInfo ( int32  input_chunk_size,
int32  num_chunks,
std::vector< ChunkInfo > *  chunk_info_out 
) const

Uses the output of the Context() functions of the network, to compute a vector of size NumComponents() + 1 indexed by component-index c, of the chunk-info at the input of each layer c, where the c+1'th element contains the chunk-info at the output of that layer.

The "input_chunk_size" is the time extent of the input. If you want to produce exactly 1 output frame per chunk, then this should equal 1 + LeftContext() + RightContext().

Definition at line 65 of file nnet-nnet.cc.

References Nnet::components_, Component::Context(), Nnet::GetComponent(), rnnlm::i, Nnet::InputDim(), rnnlm::j, KALDI_ASSERT, Nnet::LeftContext(), Nnet::NumComponents(), Nnet::OutputDim(), and Nnet::RightContext().

Referenced by NnetOnlineComputer::Compute(), NnetUpdater::ComputeForMinibatch(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), NnetUpdater::FormatInput(), NnetComputer::NnetComputer(), and NnetDiscriminativeUpdater::NnetDiscriminativeUpdater().

67  {
68  // First compute the output-chunk indices for the last component in the
69  // network. we assume that the numbering of the input starts from zero.
70  int32 output_chunk_size = input_chunk_size - LeftContext() - RightContext();
71  KALDI_ASSERT(output_chunk_size > 0);
72  std::vector<int32> current_output_inds;
73  for (int32 i = 0; i < output_chunk_size; i++)
74  current_output_inds.push_back(i + LeftContext());
75 
76  (*chunk_info_out).resize(NumComponents() + 1);
77 
78  // indexes for last component is empty, since the last component's chunk is
79  // always contiguous
80  // component's output is always contiguous
81  (*chunk_info_out)[NumComponents()] = ChunkInfo(
83  num_chunks, current_output_inds.front(),
84  current_output_inds.back());
85 
86  std::vector<int32> current_input_inds;
87  for (int32 i = NumComponents() - 1; i >= 0; i--) {
88  std::vector<int32> current_context = GetComponent(i).Context();
89  std::set<int32> current_input_ind_set;
90  for (size_t j = 0; j < current_context.size(); j++)
91  for (size_t k = 0; k < current_output_inds.size(); k++)
92  current_input_ind_set.insert(current_context[j] +
93  current_output_inds[k]);
94  current_output_inds.resize(current_input_ind_set.size());
95  std::copy(current_input_ind_set.begin(),
96  current_input_ind_set.end(),
97  current_output_inds.begin());
98 
99  // checking if the vector has contiguous data
100  // assign indexes only if the data is not contiguous
101  if (current_output_inds.size() !=
102  current_output_inds.back() - current_output_inds.front() + 1) {
103  (*chunk_info_out)[i] = ChunkInfo(GetComponent(i).InputDim(),
104  num_chunks,
105  current_output_inds);
106  } else {
107  (*chunk_info_out)[i] = ChunkInfo(GetComponent(i).InputDim(),
108  num_chunks,
109  current_output_inds.front(),
110  current_output_inds.back());
111  }
112  }
113 
114  // TODO: Make a set of components which can deal with data rearrangement.
115  // Define this set in an appropriate place so that
116  // users adding new components can simply update the list.
117  const char *dinit[] = {"SpliceComponent", "SpliceMaxComponent"};
118  std::vector< std::string > data_rearrange_components(dinit, dinit + 2);
119 
120  // Ensuring that all components until the first component capable of data
121  // rearrangement (e.g. SpliceComponent|SpliceMaxComponent) operate on
122  // contiguous chunks at the input
123  for (size_t i = 0 ; i < NumComponents() ; i++) {
124  (*chunk_info_out)[i].MakeOffsetsContiguous();
125  // Check if the current component is present in the set of components
126  // capable of data rearrangement.
127  if (std::find(data_rearrange_components.begin(),
128  data_rearrange_components.end(),
129  components_[i]->Type())
130  != data_rearrange_components.end())
131  break;
132  }
133 
134  // sanity testing for chunk_info_out vector
135  for (size_t i = 0; i < chunk_info_out->size(); i++) {
136  (*chunk_info_out)[i].Check();
137  // (*chunk_info_out)[i].ToString();
138  }
139 }
int32 InputDim() const
Dimension of the input features, e.g.
Definition: nnet-nnet.cc:36
int32 OutputDim() const
The output dimension of the network – typically the number of pdfs.
Definition: nnet-nnet.cc:31
int32 LeftContext() const
Returns the left-context summed over all the Components...
Definition: nnet-nnet.cc:42
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
virtual std::vector< int32 > Context() const
Return a vector describing the temporal context this component requires for each frame of output...
int32 RightContext() const
Returns the right-context summed over all the Components...
Definition: nnet-nnet.cc:56
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void CopyStatsFrom ( const Nnet nnet)

Copies only the statistics in layers of type NonlinearComponewnt, from this neural net, leaving everything else fixed.

Definition at line 447 of file nnet-nnet.cc.

References NonlinearComponent::Add(), Nnet::GetComponent(), rnnlm::i, KALDI_ASSERT, Nnet::NumComponents(), and NonlinearComponent::Scale().

Referenced by main().

447  {
448  KALDI_ASSERT(NumComponents() == other.NumComponents());
449  for (int32 i = 0; i < NumComponents(); i++) {
450  NonlinearComponent *nc =
451  dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
452  const NonlinearComponent *nc_other =
453  dynamic_cast<const NonlinearComponent*>(&(other.GetComponent(i)));
454  if (nc != NULL) {
455  nc->Scale(0.0);
456  nc->Add(1.0, *nc_other);
457  }
458  }
459 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void Destroy ( )

Definition at line 200 of file nnet-nnet.cc.

References Nnet::components_.

Referenced by Nnet::Init(), Nnet::operator=(), Nnet::Read(), and Nnet::~Nnet().

200  {
201  while (!components_.empty()) {
202  delete components_.back();
203  components_.pop_back();
204  }
205 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 FirstUpdatableComponent ( ) const

Returns the index of the lowest-numbered component which is updatable, or NumComponents() if none are updatable.

Definition at line 828 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and Nnet::NumComponents().

Referenced by NnetUpdater::Backprop().

828  {
829  for (int32 i = 0; i < NumComponents(); i++) {
830  if (dynamic_cast<UpdatableComponent*>(components_[i]) != NULL)
831  return i;
832  }
833  return NumComponents();
834 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
const Component & GetComponent ( int32  c) const

Definition at line 141 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by Nnet::AddNnet(), NnetStats::AddStatsFromNnet(), NnetComputer::Backprop(), NnetDiscriminativeUpdater::Backprop(), NnetUpdater::Backprop(), kaldi::nnet2::CombineNnetsA(), Nnet::ComponentDotProducts(), Nnet::ComputeChunkInfo(), kaldi::nnet2::ComputeObjfAndGradient(), FastNnetCombiner::ComputeObjfAndGradient(), NnetRescaler::ComputeRelevantIndexes(), Nnet::CopyStatsFrom(), kaldi::nnet2::FixNnet(), Nnet::GetLearningRates(), kaldi::nnet2::GetNnetStats(), Nnet::GetParameterDim(), NnetRescaler::GetTargetAvgDeriv(), kaldi::nnet2::GiveNnetCorrectTopology(), kaldi::nnet2::IndexOfSoftmaxLayer(), kaldi::nnet2::InsertComponents(), kaldi::nnet2::LimitRankParallel(), main(), Nnet::NumUpdatableComponents(), LimitRankClass::operator()(), FisherComputationClass::operator()(), NnetComputer::Propagate(), NnetDiscriminativeUpdater::Propagate(), NnetUpdater::Propagate(), NnetOnlineComputer::Propagate(), kaldi::nnet2::ReplaceLastComponents(), NnetRescaler::Rescale(), NnetRescaler::RescaleComponent(), Nnet::ResetGenerators(), Nnet::Scale(), Nnet::ScaleComponents(), Nnet::SetLearningRates(), kaldi::nnet2::SetMaxChange(), Nnet::UnVectorize(), Nnet::Vectorize(), and kaldi::nnet2::WidenNnet().

141  {
142  KALDI_ASSERT(static_cast<size_t>(component) < components_.size());
143  return *(components_[component]);
144 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
Component & GetComponent ( int32  c)

Definition at line 146 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

146  {
147  KALDI_ASSERT(static_cast<size_t>(component) < components_.size());
148  return *(components_[component]);
149 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void GetLearningRates ( VectorBase< BaseFloat > *  learning_rates) const

Get all the learning rates in the neural net (the output must have dim equal to NumUpdatableComponents()).

Definition at line 476 of file nnet-nnet.cc.

References VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, UpdatableComponent::LearningRate(), Nnet::NumComponents(), and Nnet::NumUpdatableComponents().

Referenced by main().

476  {
477  KALDI_ASSERT(learning_rates->Dim() == this->NumUpdatableComponents());
478  int32 i = 0;
479  for (int32 j = 0; j < NumComponents(); j++) {
480  const UpdatableComponent *uc =
481  dynamic_cast<const UpdatableComponent*>(&(GetComponent(j)));
482  if (uc!= NULL) {
483  (*learning_rates)(i) = uc->LearningRate();
484  i++;
485  }
486  }
487  KALDI_ASSERT(i == learning_rates->Dim());
488 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents() const
Returns the number of updatable components.
Definition: nnet-nnet.cc:413
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
int32 GetParameterDim ( ) const
virtual

Definition at line 657 of file nnet-nnet.cc.

References Nnet::GetComponent(), UpdatableComponent::GetParameterDim(), and Nnet::NumComponents().

Referenced by Nnet::Info(), main(), Nnet::UnVectorize(), and Nnet::Vectorize().

657  {
658  int32 ans = 0;
659  for (int32 c = 0; c < NumComponents(); c++) {
660  const UpdatableComponent *uc = dynamic_cast<const UpdatableComponent*>(
661  &(GetComponent(c)));
662  if (uc != NULL)
663  ans += uc->GetParameterDim();
664  }
665  return ans;
666 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
std::string Info ( ) const

Definition at line 257 of file nnet-nnet.cc.

References Nnet::components_, Nnet::GetParameterDim(), rnnlm::i, Nnet::InputDim(), Nnet::LeftContext(), Nnet::NumComponents(), Nnet::NumUpdatableComponents(), Nnet::OutputDim(), and Nnet::RightContext().

Referenced by kaldi::nnet2::DoBackprop(), AmNnet::Info(), main(), kaldi::nnet2::UnitTestNnetCompute(), and kaldi::nnet2::UnitTestNnetComputeChunked().

257  {
258  std::ostringstream ostr;
259  ostr << "num-components " << NumComponents() << std::endl;
260  ostr << "num-updatable-components " << NumUpdatableComponents() << std::endl;
261  ostr << "left-context " << LeftContext() << std::endl;
262  ostr << "right-context " << RightContext() << std::endl;
263  ostr << "input-dim " << InputDim() << std::endl;
264  ostr << "output-dim " << OutputDim() << std::endl;
265  ostr << "parameter-dim " << GetParameterDim() << std::endl;
266  for (int32 i = 0; i < NumComponents(); i++)
267  ostr << "component " << i << " : " << components_[i]->Info() << std::endl;
268  return ostr.str();
269 }
int32 InputDim() const
Dimension of the input features, e.g.
Definition: nnet-nnet.cc:36
int32 OutputDim() const
The output dimension of the network – typically the number of pdfs.
Definition: nnet-nnet.cc:31
virtual int32 GetParameterDim() const
Definition: nnet-nnet.cc:657
int32 LeftContext() const
Returns the left-context summed over all the Components...
Definition: nnet-nnet.cc:42
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
int32 NumUpdatableComponents() const
Returns the number of updatable components.
Definition: nnet-nnet.cc:413
int32 RightContext() const
Returns the right-context summed over all the Components...
Definition: nnet-nnet.cc:56
std::string Info() const
Definition: nnet-nnet.cc:257
void Init ( std::istream &  is)

Initialize from config file.

Each line of the config is either a comment line starting with whitespace then #, or it is a line that specifies one layer of the network, as accepted by Component::InitFromString(). An example non-comment line is: AffineComponent learning-rate=0.01 l2-penalty=0.001 input-dim=10 output-dim=15 param-stddev=0.1

Definition at line 281 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), KALDI_ASSERT, Component::NewFromString(), and Nnet::SetIndexes().

Referenced by kaldi::ConvertNnet1ToNnet2(), kaldi::nnet2::GenRandomNnet(), AmNnet::Init(), kaldi::nnet2::InsertComponents(), main(), and kaldi::nnet2::ReplaceLastComponents().

281  {
282  Destroy();
283  std::string line;
284  /* example config file as follows. The things in brackets specify the context
285  splicing for each layer, and after that is the info about the actual layer.
286  Imagine the input dim is 13, and the speaker dim is 40, so (13 x 9) + 40 = 527.
287  The config file might be as follows; the lines beginning with # are comments.
288 
289  # layer-type layer-options
290  AffineLayer 0.01 0.001 527 1000 0.04356
291  */
292  components_.clear();
293  while (getline(is, line)) {
294  std::istringstream line_is(line);
295  line_is >> std::ws; // Eat up whitespace.
296  if (line_is.peek() == '#' || line_is.eof()) continue; // Comment or empty.
297  Component *c = Component::NewFromString(line);
298  KALDI_ASSERT(c != NULL);
299  components_.push_back(c);
300  }
301  SetIndexes();
302  Check();
303 }
static Component * NewFromString(const std::string &initializer_line)
Initialize the Component from one line that will contain first the type, e.g.
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Init ( std::vector< Component * > *  components)

This Init method works from a vector of components.

It will take ownership of the pointers and will resize the vector to zero to avoid a chance of the caller deallocating them (but the vector itself is not deleted).

Definition at line 305 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), and Nnet::SetIndexes().

305  {
306  Destroy();
307  components_.swap(*components);
308  SetIndexes();
309  Check();
310 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
int32 InputDim ( ) const

Dimension of the input features, e.g.

13 or 40. Does not take account of frame splicing– that is done with the "chunk" mechanism, where you provide chunks of features over time.

Definition at line 36 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by NnetOnlineComputer::Compute(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), kaldi::nnet2::FormatNnetInput(), Nnet::Info(), Nnet::Nnet(), and NnetComputer::NnetComputer().

36  {
37  KALDI_ASSERT(!components_.empty());
38  return components_.front()->InputDim();
39 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 LastUpdatableComponent ( ) const

Returns the index of the highest-numbered component which is updatable, or -1 if none are updatable.

Definition at line 837 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and Nnet::NumComponents().

Referenced by main().

837  {
838  for (int32 i = NumComponents() - 1; i >= 0; i--)
839  if (dynamic_cast<UpdatableComponent*>(components_[i]) != NULL)
840  return i;
841  return -1;
842 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
int32 LeftContext ( ) const

Returns the left-context summed over all the Components...

this is the entire left-context in frames, that the network requires.

Definition at line 42 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by NnetOnlineComputer::Compute(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), DecodableAmNnet::DecodableAmNnet(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), NnetUpdater::FormatInput(), kaldi::nnet2::FormatNnetInput(), NnetDiscriminativeUpdater::GetInputFeatures(), Nnet::Info(), main(), kaldi::nnet2::NnetComputationChunked(), NnetComputer::NnetComputer(), DecodableAmNnetParallel::NumFramesReady(), kaldi::nnet2::UnitTestNnetCompute(), and kaldi::nnet2::UnitTestNnetComputeChunked().

42  {
43  KALDI_ASSERT(!components_.empty());
44  int32 ans = 0;
45  for (size_t i = 0; i < components_.size(); i++) {
46  ans += components_[i]->Context().front();
47  }
48  return -1*ans;
49  // nnet-components return left context as a non-positive integer
50  // however the nnet-update, nnet-compute expect a
51  // non-negative left context. In addition, the NnetExample also stores data
52  // left context as positive integer. To be compatible with these other classes
53  // Nnet::LeftContext() returns a non-negative left context.
54 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void LimitRankOfLastLayer ( int32  dimension)

Turns the last affine layer into two layers of the same type, with a smaller dimension in between– we're keeping the top singular values of the matrix.

Definition at line 708 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, KALDI_ERR, AffineComponent::LimitRank(), and Nnet::SetIndexes().

Referenced by main().

708  {
709  for (int32 i = components_.size() - 1; i >= 0; i--) {
710  AffineComponent *a = NULL, *b = NULL,
711  *c = dynamic_cast<AffineComponent*>(components_[i]);
712  if (c != NULL) {
713  c->LimitRank(dim, &a, &b);
714  delete c;
715  components_[i] = a;
716  components_.insert(components_.begin() + i + 1, b);
717  this->SetIndexes();
718  this->Check();
719  return;
720  }
721  }
722  KALDI_ERR << "No affine component found in neural net.";
723 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ERR
Definition: kaldi-error.h:127
void Check() const
Definition: nnet-nnet.cc:271
int32 NumComponents ( ) const
inline

Returns number of components– think of this as similar to # of layers, but e.g.

the nonlinearity and the linear part count as separate components, so the number of components will be more than the number of layers.

Definition at line 69 of file nnet-nnet.h.

References Nnet::components_.

Referenced by Nnet::AddNnet(), NnetComputer::Backprop(), NnetDiscriminativeUpdater::Backprop(), NnetUpdater::Backprop(), kaldi::nnet2::CombineNnetsA(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), NnetComputer::ComputeLastLayerDeriv(), NnetUpdater::ComputeObjfAndDeriv(), kaldi::nnet2::ComputeObjfAndGradient(), FastNnetCombiner::ComputeObjfAndGradient(), NnetRescaler::ComputeRelevantIndexes(), NnetUpdater::ComputeTotAccuracy(), Nnet::CopyStatsFrom(), Nnet::FirstUpdatableComponent(), kaldi::nnet2::FixNnet(), NnetUpdater::FormatInput(), Nnet::GetLearningRates(), kaldi::nnet2::GetNnetStats(), NnetUpdater::GetOutput(), Nnet::GetParameterDim(), kaldi::nnet2::GiveNnetCorrectTopology(), kaldi::nnet2::IndexOfSoftmaxLayer(), Nnet::Info(), kaldi::nnet2::InsertComponents(), Nnet::LastUpdatableComponent(), NnetDiscriminativeUpdater::LatticeComputations(), kaldi::nnet2::LimitRankParallel(), main(), NnetComputer::NnetComputer(), NnetOnlineComputer::NnetOnlineComputer(), Nnet::NumUpdatableComponents(), FisherComputationClass::operator()(), NnetComputer::Propagate(), NnetDiscriminativeUpdater::Propagate(), NnetOnlineComputer::Propagate(), NnetUpdater::Propagate(), kaldi::nnet2::ReplaceLastComponents(), NnetRescaler::Rescale(), Nnet::ResetGenerators(), Nnet::ResizeOutputLayer(), Nnet::Scale(), Nnet::ScaleComponents(), Nnet::ScaleLearningRates(), Nnet::SetLearningRates(), kaldi::nnet2::SetMaxChange(), Nnet::UnVectorize(), Nnet::Vectorize(), and kaldi::nnet2::WidenNnet().

69 { return components_.size(); }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumUpdatableComponents ( ) const

Returns the number of updatable components.

Definition at line 413 of file nnet-nnet.cc.

References Nnet::GetComponent(), rnnlm::i, and Nnet::NumComponents().

Referenced by Nnet::AddNnet(), Nnet::ComponentDotProducts(), FisherComputationClass::FisherComputationClass(), Nnet::GetLearningRates(), Nnet::Info(), main(), FisherComputationClass::operator()(), Nnet::ScaleComponents(), Nnet::SetLearningRates(), and kaldi::nnet2::ShrinkNnet().

413  {
414  int32 ans = 0;
415  for (int32 i = 0; i < NumComponents(); i++)
416  if (dynamic_cast<const UpdatableComponent*>(&(GetComponent(i))) != NULL)
417  ans++;
418  return ans;
419 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
Nnet & operator= ( const Nnet other)

Definition at line 247 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), rnnlm::i, and Nnet::SetIndexes().

247  {
248  Destroy();
249  components_.resize(other.components_.size());
250  for (size_t i = 0; i < other.components_.size(); i++)
251  components_[i] = other.components_[i]->Copy();
252  SetIndexes();
253  Check();
254  return *this;
255 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
int32 OutputDim ( ) const

The output dimension of the network – typically the number of pdfs.

Definition at line 31 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by Nnet::ComputeChunkInfo(), NnetUpdater::ComputeObjfAndDeriv(), Nnet::Info(), AmNnet::Init(), main(), Nnet::Nnet(), and AmNnet::NumPdfs().

31  {
32  KALDI_ASSERT(!components_.empty());
33  return components_.back()->OutputDim();
34 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Read ( std::istream &  is,
bool  binary 
)

Definition at line 175 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, Nnet::Destroy(), kaldi::ExpectToken(), kaldi::ReadBasicType(), Component::ReadNew(), and Nnet::SetIndexes().

Referenced by main(), AmNnet::Read(), and kaldi::nnet2::UnitTestNnet().

175  {
176  Destroy();
177  ExpectToken(is, binary, "<Nnet>");
178  int32 num_components;
179  ExpectToken(is, binary, "<NumComponents>");
180  ReadBasicType(is, binary, &num_components);
181  ExpectToken(is, binary, "<Components>");
182  components_.resize(num_components);
183  for (int32 c = 0; c < num_components; c++)
184  components_[c] = Component::ReadNew(is, binary);
185  ExpectToken(is, binary, "</Components>");
186  ExpectToken(is, binary, "</Nnet>");
187  SetIndexes();
188  Check();
189 }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
static Component * ReadNew(std::istream &is, bool binary)
Read component from stream.
void Check() const
Definition: nnet-nnet.cc:271
void RemoveDropout ( )

Excise any components of type DropoutComponent or AdditiveNoiseComponent.

Definition at line 497 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, KALDI_LOG, and Nnet::SetIndexes().

Referenced by main().

497  {
498  std::vector<Component*> components;
499  int32 removed = 0;
500  for (size_t i = 0; i < components_.size(); i++) {
501  if (dynamic_cast<DropoutComponent*>(components_[i]) != NULL ||
502  dynamic_cast<AdditiveNoiseComponent*>(components_[i]) != NULL) {
503  delete components_[i];
504  removed++;
505  } else {
506  components.push_back(components_[i]);
507  }
508  }
509  components_ = components;
510  if (removed > 0)
511  KALDI_LOG << "Removed " << removed << " dropout components.";
512  SetIndexes();
513  Check();
514 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
#define KALDI_LOG
Definition: kaldi-error.h:133
void RemovePreconditioning ( )

Replace any components of type AffineComponentPreconditioned with components of type AffineComponent.

Definition at line 531 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, and Nnet::SetIndexes().

Referenced by main().

531  {
532  for (size_t i = 0; i < components_.size(); i++) {
533  if (dynamic_cast<AffineComponentPreconditioned*>(components_[i]) != NULL) {
534  AffineComponent *ac = new AffineComponent(
535  *(dynamic_cast<AffineComponent*>(components_[i])));
536  delete components_[i];
537  components_[i] = ac;
538  } else if (dynamic_cast<AffineComponentPreconditionedOnline*>(
539  components_[i]) != NULL) {
540  AffineComponent *ac = new AffineComponent(
541  *(dynamic_cast<AffineComponent*>(components_[i])));
542  delete components_[i];
543  components_[i] = ac;
544  }
545  }
546  SetIndexes();
547  Check();
548 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
void ResetGenerators ( )

Definition at line 683 of file nnet-nnet.cc.

References Nnet::GetComponent(), Nnet::NumComponents(), and RandomComponent::ResetGenerator().

683  {
684  // resets random-number generators for all random
685  // components.
686  for (int32 c = 0; c < NumComponents(); c++) {
687  RandomComponent *rc = dynamic_cast<RandomComponent*>(
688  &(GetComponent(c)));
689  if (rc != NULL)
690  rc->ResetGenerator();
691  }
692 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void Resize ( int32  num_components)

Removes final components from the neural network (used for debugging).

Definition at line 490 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by main().

490  {
491  KALDI_ASSERT(new_size <= static_cast<int32>(components_.size()));
492  for (size_t i = new_size; i < components_.size(); i++)
493  delete components_[i];
494  components_.resize(new_size);
495 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void ResizeOutputLayer ( int32  new_num_pdfs)

This function is used when doing transfer learning to a new system.

It resizes the final affine and softmax components. If your system has a SumGroupComponent before the final softmax, it will be discarded.

Definition at line 356 of file nnet-nnet.cc.

References Nnet::Check(), AffineComponent::CollapseWithNext(), Nnet::components_, AffineComponent::InputDim(), KALDI_ASSERT, KALDI_ERR, Nnet::NumComponents(), AffineComponent::Resize(), and Nnet::SetIndexes().

Referenced by AmNnet::ResizeOutputLayer().

356  {
357  KALDI_ASSERT(new_num_pdfs > 0);
359  int32 nc = NumComponents();
360  SumGroupComponent *sgc =
361  dynamic_cast<SumGroupComponent*>(components_[nc - 1]);
362  if (sgc != NULL) {
363  // Remove it. We'll resize things later.
364  delete sgc;
365  components_.erase(components_.begin() + nc - 1,
366  components_.begin() + nc);
367  nc--;
368  }
369  SoftmaxComponent *sc;
370  if ((sc = dynamic_cast<SoftmaxComponent*>(components_[nc - 1])) == NULL)
371  KALDI_ERR << "Expected last component to be SoftmaxComponent.";
372 
373  // check if nc-1 has a FixedScaleComponent
374  bool has_fixed_scale_component = false;
375  int32 fixed_scale_component_index = -1;
376  int32 final_affine_component_index = nc - 2;
377  int32 softmax_component_index = nc - 1;
378  FixedScaleComponent *fsc =
379  dynamic_cast<FixedScaleComponent*>(
380  components_[final_affine_component_index]);
381  if (fsc != NULL) {
382  has_fixed_scale_component = true;
383  fixed_scale_component_index = nc - 2;
384  final_affine_component_index = nc - 3;
385  }
386  // note: it could be child class of AffineComponent.
387  AffineComponent *ac = dynamic_cast<AffineComponent*>(
388  components_[final_affine_component_index]);
389  if (ac == NULL)
390  KALDI_ERR << "Network doesn't have expected structure (didn't find final "
391  << "AffineComponent).";
392  if (has_fixed_scale_component) {
393  // collapse the fixed_scale_component with the affine_component before it
394  AffineComponent *ac_new =
395  dynamic_cast<AffineComponent*>(ac->CollapseWithNext(*fsc));
396  KALDI_ASSERT(ac_new != NULL);
397  delete fsc;
398  delete ac;
399  components_.erase(components_.begin() + fixed_scale_component_index,
400  components_.begin() + (fixed_scale_component_index + 1));
401  components_[final_affine_component_index] = ac_new;
402  ac = ac_new;
403  softmax_component_index = softmax_component_index - 1;
404  }
405  ac->Resize(ac->InputDim(), new_num_pdfs);
406  // Remove the softmax component, and replace it with a new one
407  delete components_[softmax_component_index];
408  components_[softmax_component_index] = new SoftmaxComponent(new_num_pdfs);
409  this->SetIndexes(); // used for debugging
410  this->Check();
411 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ERR
Definition: kaldi-error.h:127
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
void Check() const
Definition: nnet-nnet.cc:271
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 RightContext ( ) const

Returns the right-context summed over all the Components...

this is the entire right-context in frames, that the network requires.

Definition at line 56 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and KALDI_ASSERT.

Referenced by NnetOnlineComputer::Compute(), Nnet::ComputeChunkInfo(), NnetUpdater::ComputeForMinibatch(), DecodableAmNnet::DecodableAmNnet(), NnetOnlineComputer::Flush(), NnetRescaler::FormatInput(), NnetUpdater::FormatInput(), kaldi::nnet2::FormatNnetInput(), NnetDiscriminativeUpdater::GetInputFeatures(), Nnet::Info(), main(), kaldi::nnet2::NnetComputationChunked(), NnetComputer::NnetComputer(), DecodableAmNnetParallel::NumFramesReady(), kaldi::nnet2::UnitTestNnetCompute(), and kaldi::nnet2::UnitTestNnetComputeChunked().

56  {
57  KALDI_ASSERT(!components_.empty());
58  int32 ans = 0;
59  for (size_t i = 0; i < components_.size(); i++) {
60  ans += components_[i]->Context().back();
61  }
62  return ans;
63 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Scale ( BaseFloat  scale)

Scales all the Components with the same scale.

This applies to UpdatableComponents, and (unlike the ScaleComponents function) to SoftmaxComponents.

Definition at line 436 of file nnet-nnet.cc.

References Nnet::GetComponent(), rnnlm::i, Nnet::NumComponents(), UpdatableComponent::Scale(), and NonlinearComponent::Scale().

Referenced by main().

436  {
437  for (int32 i = 0; i < NumComponents(); i++) {
438  UpdatableComponent *uc =
439  dynamic_cast<UpdatableComponent*>(&(GetComponent(i)));
440  if (uc != NULL) uc->Scale(scale);
441  NonlinearComponent *nc =
442  dynamic_cast<NonlinearComponent*>(&(GetComponent(i)));
443  if (nc != NULL) nc->Scale(scale);
444  }
445 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void ScaleComponents ( const VectorBase< BaseFloat > &  scales)

Scales the parameters of each of the updatable components.

Here, scale_params is a vector of size equal to NumUpdatableComponents()

Definition at line 421 of file nnet-nnet.cc.

References VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), Nnet::NumUpdatableComponents(), and UpdatableComponent::Scale().

Referenced by kaldi::nnet2::CombineNnets(), FastNnetCombiner::CombineNnets(), kaldi::nnet2::ComputeObjfAndGradient(), kaldi::nnet2::GetUpdateDirection(), main(), and kaldi::nnet2::ShrinkNnet().

421  {
422  KALDI_ASSERT(scale_params.Dim() == this->NumUpdatableComponents());
423  int32 i = 0;
424  for (int32 j = 0; j < NumComponents(); j++) {
425  UpdatableComponent *uc =
426  dynamic_cast<UpdatableComponent*>(&(GetComponent(j)));
427  if (uc!= NULL) {
428  uc->Scale(scale_params(i));
429  i++;
430  }
431  }
432  KALDI_ASSERT(i == scale_params.Dim());
433 }
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents() const
Returns the number of updatable components.
Definition: nnet-nnet.cc:413
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void ScaleLearningRates ( BaseFloat  factor)

Scale all the learning rates in the neural net by this factor.

Definition at line 313 of file nnet-nnet.cc.

References Nnet::components_, KALDI_LOG, UpdatableComponent::LearningRate(), Nnet::NumComponents(), and UpdatableComponent::SetLearningRate().

Referenced by main().

313  {
314  std::ostringstream ostr;
315  for (int32 c = 0; c < NumComponents(); c++) {
316  UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[c]);
317  if (uc != NULL) { // Updatable component...
318  uc->SetLearningRate(uc->LearningRate() * factor);
319  ostr << uc->LearningRate() << " ";
320  }
321  }
322  KALDI_LOG << "Scaled learning rates by " << factor
323  << ", new learning rates are "
324  << ostr.str();
325 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_LOG
Definition: kaldi-error.h:133
void ScaleLearningRates ( std::map< std::string, BaseFloat scale_factors)

Scale all the learning rates in the neural net by the factors indexed by the type of component.

Definition at line 327 of file nnet-nnet.cc.

References Nnet::components_, KALDI_LOG, UpdatableComponent::LearningRate(), Nnet::NumComponents(), UpdatableComponent::SetLearningRate(), and Component::Type().

327  {
328  std::ostringstream ostr;
329  for (int32 c = 0; c < NumComponents(); c++) {
330  UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[c]);
331  if (uc != NULL) { // Updatable component...
332  // check if scaling factor was specified for a component of this type
333  std::map<std::string, BaseFloat>::const_iterator lr_iterator =
334  scale_factors.find(uc->Type());
335  if (lr_iterator != scale_factors.end()) {
336  uc->SetLearningRate(uc->LearningRate() * lr_iterator->second);
337  ostr << uc->LearningRate() << " ";
338  }
339  }
340  }
341  KALDI_LOG << "Scaled learning rates by component-type specific factor, "
342  << "new learning rates are "
343  << ostr.str();
344 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_LOG
Definition: kaldi-error.h:133
void SetComponent ( int32  c,
Component component 
)

Sets the c'th component to "component", taking ownership of the pointer and deleting the corresponding one that we own.

Definition at line 649 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, KALDI_ASSERT, and Nnet::SetIndexes().

649  {
650  KALDI_ASSERT(static_cast<size_t>(c) < components_.size());
651  delete components_[c];
652  components_[c] = component;
653  SetIndexes();
654  Check(); // Check that all the dimensions still match up.
655 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SetDropoutScale ( BaseFloat  scale)

Calls SetDropoutScale for all the dropout nodes.

Definition at line 516 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, KALDI_LOG, and DropoutComponent::SetDropoutScale().

Referenced by main().

516  {
517  size_t n_set = 0;
518  for (size_t i = 0; i < components_.size(); i++) {
519  DropoutComponent *dc =
520  dynamic_cast<DropoutComponent*>(components_[i]);
521  if (dc != NULL) {
522  dc->SetDropoutScale(scale);
523  n_set++;
524  }
525  }
526  KALDI_LOG << "Set dropout scale to " << scale
527  << " for " << n_set << " components.";
528 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
#define KALDI_LOG
Definition: kaldi-error.h:133
void SetIndexes ( )

Sets the index_ values of the components.

Definition at line 725 of file nnet-nnet.cc.

References Nnet::components_, and rnnlm::i.

Referenced by Nnet::Append(), Nnet::Collapse(), Nnet::Init(), Nnet::LimitRankOfLastLayer(), Nnet::Nnet(), Nnet::operator=(), Nnet::Read(), Nnet::RemoveDropout(), Nnet::RemovePreconditioning(), Nnet::ResizeOutputLayer(), Nnet::SetComponent(), and Nnet::SwitchToOnlinePreconditioning().

725  {
726  for (size_t i = 0; i < components_.size(); i++)
727  components_[i]->SetIndex(i);
728 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void SetLearningRates ( BaseFloat  learning_rates)

Set all the learning rates in the neural net to this value.

Definition at line 346 of file nnet-nnet.cc.

References Nnet::components_, KALDI_LOG, Nnet::NumComponents(), and UpdatableComponent::SetLearningRate().

Referenced by main().

346  {
347  for (int32 c = 0; c < NumComponents(); c++) {
348  UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[c]);
349  if (uc != NULL) { // Updatable component...
350  uc->SetLearningRate(learning_rate);
351  }
352  }
353  KALDI_LOG << "Set learning rates to " << learning_rate;
354 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_LOG
Definition: kaldi-error.h:133
void SetLearningRates ( const VectorBase< BaseFloat > &  learning_rates)

Set all the learning rates in the neural net to these values (one for each updatable layer).

Definition at line 461 of file nnet-nnet.cc.

References VectorBase< Real >::Dim(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, VectorBase< Real >::Min(), Nnet::NumComponents(), Nnet::NumUpdatableComponents(), and UpdatableComponent::SetLearningRate().

461  {
462  KALDI_ASSERT(learning_rates.Dim() == this->NumUpdatableComponents());
463  KALDI_ASSERT(learning_rates.Min() >= 0.0); // we allow zero learning rate.
464  int32 i = 0;
465  for (int32 j = 0; j < NumComponents(); j++) {
466  UpdatableComponent *uc =
467  dynamic_cast<UpdatableComponent*>(&(GetComponent(j)));
468  if (uc!= NULL) {
469  uc->SetLearningRate(learning_rates(i));
470  i++;
471  }
472  }
473  KALDI_ASSERT(i == learning_rates.Dim());
474 }
Real Min() const
Returns the minimum value of any element, or +infinity for the empty vector.
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents() const
Returns the number of updatable components.
Definition: nnet-nnet.cc:413
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
void SetZero ( bool  treat_as_gradient)

Definition at line 151 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, NonlinearComponent::Scale(), and UpdatableComponent::SetZero().

Referenced by kaldi::nnet2::ComputeNnetGradient(), kaldi::nnet2::ComputeObjfAndGradient(), FastNnetCombiner::ComputeObjfAndGradient(), DiscTrainParallelClass::DiscTrainParallelClass(), DoBackpropParallelClass::DoBackpropParallelClass(), main(), and FisherComputationClass::operator()().

151  {
152  for (size_t i = 0; i < components_.size(); i++) {
153  UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(components_[i]);
154  if (uc != NULL) uc->SetZero(treat_as_gradient);
155  NonlinearComponent *nc = dynamic_cast<NonlinearComponent*>(components_[i]);
156  if (nc != NULL) nc->Scale(0.0);
157  }
158 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void SwitchToOnlinePreconditioning ( int32  rank_in,
int32  rank_out,
int32  update_period,
BaseFloat  num_samples_history,
BaseFloat  alpha 
)

Replaces any components of type AffineComponent or derived classes, with components of type AffineComponentPreconditionedOnline.

E.g. rank_in = 20, rank_out = 80, num_samples_history = 2000.0, alpha = 4.0

Definition at line 551 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, KALDI_LOG, and Nnet::SetIndexes().

Referenced by main().

554  {
555  int32 switched = 0;
556  for (size_t i = 0; i < components_.size(); i++) {
557  if (dynamic_cast<AffineComponent*>(components_[i]) != NULL) {
558  AffineComponentPreconditionedOnline *ac =
559  new AffineComponentPreconditionedOnline(
560  *(dynamic_cast<AffineComponent*>(components_[i])),
561  rank_in, rank_out, update_period, num_samples_history, alpha);
562  delete components_[i];
563  components_[i] = ac;
564  switched++;
565  }
566  }
567  KALDI_LOG << "Switched " << switched << " components to use online "
568  << "preconditioning, with (input, output) rank = "
569  << rank_in << ", " << rank_out << " and num_samples_history = "
570  << num_samples_history;
571  SetIndexes();
572  Check();
573 }
void SetIndexes()
Sets the index_ values of the components.
Definition: nnet-nnet.cc:725
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void Check() const
Definition: nnet-nnet.cc:271
#define KALDI_LOG
Definition: kaldi-error.h:133
void UnVectorize ( const VectorBase< BaseFloat > &  params)
virtual

Definition at line 694 of file nnet-nnet.cc.

References Nnet::GetComponent(), Nnet::GetParameterDim(), UpdatableComponent::GetParameterDim(), KALDI_ASSERT, Nnet::NumComponents(), VectorBase< Real >::Range(), and UpdatableComponent::UnVectorize().

694  {
695  int32 offset = 0;
696  for (int32 c = 0; c < NumComponents(); c++) {
697  UpdatableComponent *uc = dynamic_cast<UpdatableComponent*>(
698  &(GetComponent(c)));
699  if (uc != NULL) {
700  int32 size = uc->GetParameterDim();
701  uc->UnVectorize(params.Range(offset, size));
702  offset += size;
703  }
704  }
705  KALDI_ASSERT(offset == GetParameterDim());
706 }
virtual int32 GetParameterDim() const
Definition: nnet-nnet.cc:657
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void Vectorize ( VectorBase< BaseFloat > *  params) const
virtual

Definition at line 668 of file nnet-nnet.cc.

References Nnet::GetComponent(), Nnet::GetParameterDim(), UpdatableComponent::GetParameterDim(), KALDI_ASSERT, Nnet::NumComponents(), and UpdatableComponent::Vectorize().

668  {
669  int32 offset = 0;
670  for (int32 c = 0; c < NumComponents(); c++) {
671  const UpdatableComponent *uc = dynamic_cast<const UpdatableComponent*>(
672  &(GetComponent(c)));
673  if (uc != NULL) {
674  int32 size = uc->GetParameterDim();
675  SubVector<BaseFloat> temp(*params, offset, size);
676  uc->Vectorize(&temp);
677  offset += size;
678  }
679  }
680  KALDI_ASSERT(offset == GetParameterDim());
681 }
virtual int32 GetParameterDim() const
Definition: nnet-nnet.cc:657
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void Write ( std::ostream &  os,
bool  binary 
) const

Definition at line 160 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, kaldi::WriteBasicType(), and kaldi::WriteToken().

Referenced by kaldi::nnet2::UnitTestNnet(), and AmNnet::Write().

160  {
161  Check();
162  WriteToken(os, binary, "<Nnet>");
163  int32 num_components = components_.size();
164  WriteToken(os, binary, "<NumComponents>");
165  WriteBasicType(os, binary, num_components);
166  WriteToken(os, binary, "<Components>");
167  for (int32 c = 0; c < num_components; c++) {
168  components_[c]->Write(os, binary);
169  if (!binary) os << std::endl;
170  }
171  WriteToken(os, binary, "</Components>");
172  WriteToken(os, binary, "</Nnet>");
173 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
void Check() const
Definition: nnet-nnet.cc:271
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
void ZeroStats ( )

Definition at line 192 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, and NonlinearComponent::Scale().

Referenced by main().

192  {
193  for (size_t i = 0; i < components_.size(); i++) {
194  NonlinearComponent *nonlinear_component =
195  dynamic_cast<NonlinearComponent*>(components_[i]);
196  if (nonlinear_component != NULL)
197  nonlinear_component->Scale(0.0); // Zero the stats this way.
198  }
199 }
std::vector< Component * > components_
Definition: nnet-nnet.h:290

Friends And Related Function Documentation

friend class DecodableNnet
friend

Definition at line 288 of file nnet-nnet.h.

friend class NnetUpdater
friend

Definition at line 287 of file nnet-nnet.h.

Member Data Documentation


The documentation for this class was generated from the following files: