Nnet Class Reference

#include <nnet-nnet.h>

Collaboration diagram for Nnet:

Public Member Functions
	Nnet ()
	~Nnet ()
	Nnet (const Nnet &other)
Nnet &	operator= (const Nnet &other)
void	Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward pass through the network,. More...
void	Backpropagate (const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
	Perform backward pass through the network,. More...
void	Feedforward (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward pass through the network (with 2 swapping buffers),. More...
int32	InputDim () const
	Dimensionality on network input (input feature dim.),. More...
int32	OutputDim () const
	Dimensionality of network outputs (posteriors | bn-features | etc.),. More...
int32	NumComponents () const
	Returns the number of 'Components' which form the NN. More...
const Component &	GetComponent (int32 c) const
	Component accessor,. More...
Component &	GetComponent (int32 c)
	Component accessor,. More...
const Component &	GetLastComponent () const
	LastComponent accessor,. More...
Component &	GetLastComponent ()
	LastComponent accessor,. More...
void	ReplaceComponent (int32 c, const Component &comp)
	Replace c'th component in 'this' Nnet (deep copy),. More...
void	SwapComponent (int32 c, Component **comp)
	Swap c'th component with the pointer,. More...
void	AppendComponent (const Component &comp)
	Append Component to 'this' instance of Nnet (deep copy),. More...
void	AppendComponentPointer (Component *dynamically_allocated_comp)
	Append Component* to 'this' instance of Nnet by a shallow copy ('this' instance of Nnet over-takes the ownership of the pointer). More...
void	AppendNnet (const Nnet &nnet_to_append)
	Append other Nnet to the 'this' Nnet (copy all its components),. More...
void	RemoveComponent (int32 c)
	Remove c'th component,. More...
void	RemoveLastComponent ()
	Remove the last of the Components,. More...
const std::vector< CuMatrix< BaseFloat > > &	PropagateBuffer () const
	Access to the forward-pass buffers. More...
const std::vector< CuMatrix< BaseFloat > > &	BackpropagateBuffer () const
	Access to the backward-pass buffers. More...
int32	NumParams () const
	Get the number of parameters in the network,. More...
void	GetGradient (Vector< BaseFloat > *gradient) const
	Get the gradient stored in the network,. More...
void	GetParams (Vector< BaseFloat > *params) const
	Get the network weights in a supervector,. More...
void	SetParams (const VectorBase< BaseFloat > &params)
	Set the network weights from a supervector,. More...
void	SetDropoutRate (BaseFloat r)
	Set the dropout rate. More...
void	ResetStreams (const std::vector< int32 > &stream_reset_flag)
	Reset streams in multi-stream training,. More...
void	SetSeqLengths (const std::vector< int32 > &sequence_lengths)
	Set sequence length in LSTM multi-stream training,. More...
void	Init (const std::string &proto_file)
	Initialize the Nnet from the prototype,. More...
void	Read (const std::string &rxfilename)
	Read Nnet from 'rxfilename',. More...
void	Read (std::istream &in, bool binary)
	Read Nnet from 'istream',. More...
void	Write (const std::string &wxfilename, bool binary) const
	Write Nnet to 'wxfilename',. More...
void	Write (std::ostream &out, bool binary) const
	Write Nnet to 'ostream',. More...
std::string	Info () const
	Create string with human readable description of the nnet,. More...
std::string	InfoGradient (bool header=true) const
	Create string with per-component gradient statistics,. More...
std::string	InfoPropagate (bool header=true) const
	Create string with propagation-buffer statistics,. More...
std::string	InfoBackPropagate (bool header=true) const
	Create string with back-propagation-buffer statistics,. More...
void	Check () const
	Consistency check,. More...
void	Destroy ()
	Relese the memory,. More...
void	SetTrainOptions (const NnetTrainOptions &opts)
	Set hyper-parameters of the training (pushes to all UpdatableComponents),. More...
const NnetTrainOptions &	GetTrainOptions () const
	Get training hyper-parameters from the network,. More...

Private Attributes
std::vector< Component * >	components_
	Vector which contains all the components composing the neural network, the components are for example: AffineTransform, Sigmoid, Softmax. More...
std::vector< CuMatrix< BaseFloat > >	propagate_buf_
	Buffers for forward pass (on demand initialization),. More...
std::vector< CuMatrix< BaseFloat > >	backpropagate_buf_
	Buffers for backward pass (on demand initialization),. More...
NnetTrainOptions	opts_
	Option class with hyper-parameters passed to UpdatableComponent(s) More...

Detailed Description

Definition at line 37 of file nnet-nnet.h.

Constructor & Destructor Documentation

Nnet() [1/2]

Nnet

(

)

Definition at line 31 of file nnet-nnet.cc.

           {
}

~Nnet()

~Nnet

(

)

Definition at line 34 of file nnet-nnet.cc.

References Nnet::Destroy().

            {
  Destroy();
}

Nnet() [2/2]

Nnet

(

const Nnet &

other

)

Definition at line 38 of file nnet-nnet.cc.

References Nnet::backpropagate_buf_, Nnet::Check(), Nnet::components_, Component::Copy(), Nnet::GetComponent(), rnnlm::i, Nnet::NumComponents(), Nnet::opts_, Nnet::propagate_buf_, and Nnet::SetTrainOptions().

                            {
  // copy the components
  for (int32 i = 0; i < other.NumComponents(); i++) {
    components_.push_back(other.GetComponent(i).Copy());
  }
  // create empty buffers
  propagate_buf_.resize(NumComponents()+1);
  backpropagate_buf_.resize(NumComponents()+1);
  // copy train opts
  SetTrainOptions(other.opts_);
  Check();
}

Member Function Documentation

AppendComponent()

void AppendComponent

(

const Component &

comp

)

Append Component to 'this' instance of Nnet (deep copy),.

Definition at line 182 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, and Component::Copy().

Referenced by Nnet::AppendNnet(), main(), and Nnet::NumComponents().

                                                {
  components_.push_back(comp.Copy());  // append,
  Check();
}

AppendComponentPointer()

void AppendComponentPointer

(

Component *

dynamically_allocated_comp

)

Append Component* to 'this' instance of Nnet by a shallow copy ('this' instance of Nnet over-takes the ownership of the pointer).

Definition at line 187 of file nnet-nnet.cc.

References Nnet::Check(), and Nnet::components_.

Referenced by Nnet::Init(), main(), Nnet::NumComponents(), and Nnet::Read().

                                                                       {
  components_.push_back(dynamically_allocated_comp);  // append,
  Check();
}

AppendNnet()

void AppendNnet

(

const Nnet &

nnet_to_append

)

Append other Nnet to the 'this' Nnet (copy all its components),.

Definition at line 192 of file nnet-nnet.cc.

References Nnet::AppendComponent(), Nnet::Check(), Nnet::GetComponent(), rnnlm::i, and Nnet::NumComponents().

Referenced by main(), and Nnet::NumComponents().

                                       {
  for (int32 i = 0; i < other.NumComponents(); i++) {
    AppendComponent(other.GetComponent(i));
  }
  Check();
}

Backpropagate()

void Backpropagate	(	const CuMatrixBase< BaseFloat > &	out_diff,
		CuMatrix< BaseFloat > *	in_diff
	)

Perform backward pass through the network,.

Error back-propagation through the network, (from last component to first).

Definition at line 96 of file nnet-nnet.cc.

References Nnet::backpropagate_buf_, Nnet::components_, rnnlm::i, KALDI_ASSERT, Nnet::NumComponents(), Nnet::propagate_buf_, and UpdatableComponent::Update().

Referenced by MultiBasisComponent::BackpropagateFnc(), and main().

                                                       {
  // Copy the derivative in case of empty network,
  if (NumComponents() == 0) {
    (*in_diff) = out_diff;  // copy,
    return;
  }
  // We need C+1 buffers,
  KALDI_ASSERT(static_cast<int32>(propagate_buf_.size()) == NumComponents()+1);
  if (backpropagate_buf_.size() != NumComponents()+1) {
    backpropagate_buf_.resize(NumComponents()+1);
  }
  // Copy 'out_diff' to last buffer,
  backpropagate_buf_[NumComponents()] = out_diff;
  // Loop from last Component to the first,
  for (int32 i = NumComponents()-1; i >= 0; i--) {
    // Backpropagate through 'Component',
    components_[i]->Backpropagate(propagate_buf_[i],
                                  propagate_buf_[i+1],
                                  backpropagate_buf_[i+1],
                                  &backpropagate_buf_[i]);
    // Update 'Component' (if applicable),
    if (components_[i]->IsUpdatable()) {
      UpdatableComponent* uc =
        dynamic_cast<UpdatableComponent*>(components_[i]);
      uc->Update(propagate_buf_[i], backpropagate_buf_[i+1]);
    }
  }
  // Export the derivative (if applicable),
  if (NULL != in_diff) {
    (*in_diff) = backpropagate_buf_[0];
  }
}

BackpropagateBuffer()

const std::vector<CuMatrix<BaseFloat> >& BackpropagateBuffer ( ) const

inline

Access to the backward-pass buffers.

Definition at line 109 of file nnet-nnet.h.

References Nnet::backpropagate_buf_, Nnet::Check(), Nnet::Destroy(), Nnet::GetGradient(), Nnet::GetParams(), Nnet::Info(), Nnet::InfoBackPropagate(), Nnet::InfoGradient(), Nnet::InfoPropagate(), Nnet::Init(), Nnet::NumParams(), Nnet::Read(), Nnet::ResetStreams(), Nnet::SetDropoutRate(), Nnet::SetParams(), Nnet::SetSeqLengths(), Nnet::SetTrainOptions(), and Nnet::Write().

                                                                     {
    return backpropagate_buf_;
  }

Check()

void Check

(

)

const

Consistency check,.

Definition at line 467 of file nnet-nnet.cc.

References Nnet::components_, Nnet::GetParams(), rnnlm::i, KALDI_ASSERT, KALDI_ERR, KALDI_ISINF, KALDI_ISNAN, VectorBase< Real >::Sum(), and Component::TypeToMarker().

Referenced by Nnet::AppendComponent(), Nnet::AppendComponentPointer(), Nnet::AppendNnet(), Nnet::BackpropagateBuffer(), Nnet::Init(), Nnet::Nnet(), Nnet::operator=(), Nnet::Read(), Nnet::RemoveComponent(), Nnet::ReplaceComponent(), Nnet::SwapComponent(), and Nnet::Write().

                       {
  // check dims,
  for (size_t i = 0; i + 1 < components_.size(); i++) {
    KALDI_ASSERT(components_[i] != NULL);
    int32 output_dim = components_[i]->OutputDim(),
      next_input_dim = components_[i+1]->InputDim();
    // show error message,
    if (output_dim != next_input_dim) {
      KALDI_ERR << "Component dimension mismatch!"
                << " Output dim of [" << i << "] "
                << Component::TypeToMarker(components_[i]->GetType())
                << " is " << output_dim << ". "
                << "Input dim of next [" << i+1 << "] "
                << Component::TypeToMarker(components_[i+1]->GetType())
                << " is " << next_input_dim << ".";
    }
  }
  // check for nan/inf in network weights,
  Vector<BaseFloat> weights;
  GetParams(&weights);
  BaseFloat sum = weights.Sum();
  if (KALDI_ISINF(sum)) {
    KALDI_ERR << "'inf' in network parameters "
              << "(weight explosion, need lower learning rate?)";
  }
  if (KALDI_ISNAN(sum)) {
    KALDI_ERR << "'nan' in network parameters (need lower learning rate?)";
  }
}

Destroy()

void Destroy

(

)

Relese the memory,.

Definition at line 498 of file nnet-nnet.cc.

References Nnet::backpropagate_buf_, Nnet::components_, rnnlm::i, Nnet::NumComponents(), and Nnet::propagate_buf_.

Referenced by Nnet::BackpropagateBuffer(), Nnet::operator=(), and Nnet::~Nnet().

                   {
  for (int32 i = 0; i < NumComponents(); i++) {
    delete components_[i];
  }
  components_.resize(0);
  propagate_buf_.resize(0);
  backpropagate_buf_.resize(0);
}

Feedforward()

void Feedforward	(	const CuMatrixBase< BaseFloat > &	in,
		CuMatrix< BaseFloat > *	out
	)

Perform forward pass through the network (with 2 swapping buffers),.

Definition at line 131 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, KALDI_ASSERT, Nnet::NumComponents(), and CuMatrix< Real >::Swap().

Referenced by main().

                                                 {
  KALDI_ASSERT(NULL != out);
  (*out) = in;  // works even with 0 components,
  CuMatrix<BaseFloat> tmp_in;
  for (int32 i = 0; i < NumComponents(); i++) {
    out->Swap(&tmp_in);
    components_[i]->Propagate(tmp_in, out);
  }
}

GetComponent() [1/2]

const Component & GetComponent

(

int32

c

)

const

Component accessor,.

Definition at line 153 of file nnet-nnet.cc.

References Nnet::components_.

Referenced by Nnet::AppendNnet(), kaldi::ConvertNnet1ToNnet2(), Rbm::InitData(), main(), Nnet::Nnet(), Nnet::NumComponents(), Nnet::operator=(), Nnet::ResetStreams(), Nnet::SetDropoutRate(), Nnet::SetSeqLengths(), and Nnet::SetTrainOptions().

                                                 {
  return *(components_.at(c));
}

GetComponent() [2/2]

Component & GetComponent

(

int32

c

)

Component accessor,.

Definition at line 157 of file nnet-nnet.cc.

References Nnet::components_.

                                     {
  return *(components_.at(c));
}

GetGradient()

void GetGradient

(

Vector< BaseFloat > *

gradient

)

const

Get the gradient stored in the network,.

Definition at line 221 of file nnet-nnet.cc.

References Nnet::components_, UpdatableComponent::GetGradient(), rnnlm::i, KALDI_ASSERT, Nnet::NumParams(), UpdatableComponent::NumParams(), VectorBase< Real >::Range(), and Vector< Real >::Resize().

Referenced by Nnet::BackpropagateBuffer().

                                                        {
  gradient->Resize(NumParams());
  int32 pos = 0;
  // loop over Components,
  for (int32 i = 0; i < components_.size(); i++) {
    if (components_[i]->IsUpdatable()) {
      UpdatableComponent& c =
        dynamic_cast<UpdatableComponent&>(*components_[i]);
      SubVector<BaseFloat> grad_range(gradient->Range(pos, c.NumParams()));
      c.GetGradient(&grad_range);  // getting gradient,
      pos += c.NumParams();
    }
  }
  KALDI_ASSERT(pos == NumParams());
}

GetLastComponent() [1/2]

const Component & GetLastComponent

(

)

const

LastComponent accessor,.

Definition at line 161 of file nnet-nnet.cc.

References Nnet::components_, and Nnet::NumComponents().

Referenced by main(), and Nnet::NumComponents().

                                              {
  return *(components_.at(NumComponents()-1));
}

GetLastComponent() [2/2]

Component & GetLastComponent

(

)

LastComponent accessor,.

Definition at line 165 of file nnet-nnet.cc.

References Nnet::components_, and Nnet::NumComponents().

                                  {
  return *(components_.at(NumComponents()-1));
}

GetParams()

void GetParams

(

Vector< BaseFloat > *

params

)

const

Get the network weights in a supervector,.

Definition at line 237 of file nnet-nnet.cc.

References Nnet::components_, UpdatableComponent::GetParams(), rnnlm::i, KALDI_ASSERT, Nnet::NumParams(), UpdatableComponent::NumParams(), VectorBase< Real >::Range(), and Vector< Real >::Resize().

Referenced by Nnet::BackpropagateBuffer(), Nnet::Check(), and MultiBasisComponent::GetParams().

                                                    {
  params->Resize(NumParams());
  int32 pos = 0;
  // loop over Components,
  for (int32 i = 0; i < components_.size(); i++) {
    if (components_[i]->IsUpdatable()) {
      UpdatableComponent& c =
        dynamic_cast<UpdatableComponent&>(*components_[i]);
      SubVector<BaseFloat> params_range(params->Range(pos, c.NumParams()));
      c.GetParams(&params_range);  // getting params,
      pos += c.NumParams();
    }
  }
  KALDI_ASSERT(pos == NumParams());
}

GetTrainOptions()

const NnetTrainOptions& GetTrainOptions ( ) const

inline

Get training hyper-parameters from the network,.

Definition at line 163 of file nnet-nnet.h.

References Nnet::opts_.

                                                  {
    return opts_;
  }

Info()

std::string Info

(

)

const

Create string with human readable description of the nnet,.

Definition at line 386 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, Nnet::InputDim(), Nnet::NumComponents(), Nnet::NumParams(), Nnet::OutputDim(), and Component::TypeToMarker().

Referenced by Nnet::BackpropagateBuffer(), MultiBasisComponent::Info(), and main().

                           {
  // global info
  std::ostringstream ostr;
  ostr << "num-components " << NumComponents() << std::endl;
  if (NumComponents() == 0)
    return ostr.str();
  ostr << "input-dim " << InputDim() << std::endl;
  ostr << "output-dim " << OutputDim() << std::endl;
  ostr << "number-of-parameters " << static_cast<float>(NumParams())/1e6
       << " millions" << std::endl;
  // topology & weight stats
  for (int32 i = 0; i < NumComponents(); i++) {
    ostr << "component " << i+1 << " : "
         << Component::TypeToMarker(components_[i]->GetType())
         << ", input-dim " << components_[i]->InputDim()
         << ", output-dim " << components_[i]->OutputDim()
         << ", " << components_[i]->Info() << std::endl;
  }
  return ostr.str();
}

InfoBackPropagate()

std::string InfoBackPropagate

(

bool

header = true

)

const

Create string with back-propagation-buffer statistics,.

Definition at line 443 of file nnet-nnet.cc.

References Nnet::backpropagate_buf_, Nnet::components_, rnnlm::i, Component::kMultiBasisComponent, Component::kParallelComponent, kaldi::nnet1::MomentStatistics(), Nnet::NumComponents(), and Component::TypeToMarker().

Referenced by Nnet::BackpropagateBuffer(), MultiBasisComponent::InfoBackPropagate(), and main().

                                                   {
  std::ostringstream ostr;
  // forward-pass buffer stats
  if (header) ostr << "\n### BACKWARD PROPAGATION BUFFER CONTENT :\n";
  ostr << "[0] diff of <Input> " << MomentStatistics(backpropagate_buf_[0])
       << std::endl;
  for (int32 i = 0; i < NumComponents(); i++) {
    ostr << "["<<1+i<< "] diff-output of "
         << Component::TypeToMarker(components_[i]->GetType())
         << MomentStatistics(backpropagate_buf_[i+1]) << std::endl;
    // nested networks too...
    if (Component::kParallelComponent == components_[i]->GetType()) {
      ostr <<
        dynamic_cast<ParallelComponent*>(components_[i])->InfoBackPropagate();
    }
    if (Component::kMultiBasisComponent == components_[i]->GetType()) {
      ostr << dynamic_cast<MultiBasisComponent*>(components_[i])->InfoBackPropagate();
    }
  }
  if (header) ostr << "### END BACKWARD\n\n";
  return ostr.str();
}

InfoGradient()

std::string InfoGradient

(

bool

header = true

)

const

Create string with per-component gradient statistics,.

Definition at line 407 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, Nnet::NumComponents(), and Component::TypeToMarker().

Referenced by Nnet::BackpropagateBuffer(), MultiBasisComponent::InfoGradient(), and main().

                                              {
  std::ostringstream ostr;
  // gradient stats
  if (header) ostr << "\n### GRADIENT STATS :\n";
  for (int32 i = 0; i < NumComponents(); i++) {
    ostr << "Component " << i+1 << " : "
         << Component::TypeToMarker(components_[i]->GetType())
         << ", " << components_[i]->InfoGradient() << std::endl;
  }
  if (header) ostr << "### END GRADIENT\n";
  return ostr.str();
}

InfoPropagate()

std::string InfoPropagate

(

bool

header = true

)

const

Create string with propagation-buffer statistics,.

Definition at line 420 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, Component::kMultiBasisComponent, Component::kParallelComponent, kaldi::nnet1::MomentStatistics(), Nnet::NumComponents(), Nnet::propagate_buf_, and Component::TypeToMarker().

Referenced by Nnet::BackpropagateBuffer(), MultiBasisComponent::InfoPropagate(), and main().

                                               {
  std::ostringstream ostr;
  // forward-pass buffer stats
  if (header) ostr << "\n### FORWARD PROPAGATION BUFFER CONTENT :\n";
  ostr << "[0] output of <Input> " << MomentStatistics(propagate_buf_[0])
       << std::endl;
  for (int32 i = 0; i < NumComponents(); i++) {
    ostr << "[" << 1+i << "] output of "
         << Component::TypeToMarker(components_[i]->GetType())
         << MomentStatistics(propagate_buf_[i+1]) << std::endl;
    // nested networks too...
    if (Component::kParallelComponent == components_[i]->GetType()) {
      ostr <<
        dynamic_cast<ParallelComponent*>(components_[i])->InfoPropagate();
    }
    if (Component::kMultiBasisComponent == components_[i]->GetType()) {
      ostr << dynamic_cast<MultiBasisComponent*>(components_[i])->InfoPropagate();
    }
  }
  if (header) ostr << "### END FORWARD\n";
  return ostr.str();
}

Init()

void Init

(

const std::string &

proto_file

)

Initialize the Nnet from the prototype,.

Definition at line 301 of file nnet-nnet.cc.

References Nnet::AppendComponentPointer(), Nnet::Check(), Input::Close(), Component::Init(), KALDI_ASSERT, KALDI_VLOG, and Input::Stream().

Referenced by Nnet::BackpropagateBuffer(), MultiBasisComponent::InitData(), ParallelComponent::InitData(), and main().

                                           {
  Input in(proto_file);
  std::istream &is = in.Stream();
  std::string proto_line, token;

  // Initialize from the prototype, where each line
  // contains the description for one component.
  while (is >> std::ws, !is.eof()) {
    KALDI_ASSERT(is.good());

    // get a line from the proto file,
    std::getline(is, proto_line);
    if (proto_line == "") continue;
    KALDI_VLOG(1) << proto_line;

    // get the 1st token from the line,
    std::istringstream(proto_line) >> std::ws >> token;
    // ignore these tokens:
    if (token == "<NnetProto>" || token == "</NnetProto>") continue;

    // create new component, append to Nnet,
    this->AppendComponentPointer(Component::Init(proto_line+"\n"));
  }
  // cleanup
  in.Close();
  Check();
}

InputDim()

int32 InputDim

(

)

const

Dimensionality on network input (input feature dim.),.

Definition at line 148 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by MultiBasisComponent::BackpropagateFnc(), Nnet::Info(), MultiBasisComponent::InitData(), Rbm::InitData(), main(), MultiBasisComponent::PropagateFnc(), and MultiBasisComponent::ReadData().

                           {
  KALDI_ASSERT(!components_.empty());
  return components_.front()->InputDim();
}

NumComponents()

int32 NumComponents ( ) const

inline

Returns the number of 'Components' which form the NN.

Typically a NN layer is composed of 2 components: the <AffineTransform> with trainable parameters and a non-linearity like <Sigmoid> or <Softmax>. Usually there are 2x more Components than the NN layers.

Definition at line 66 of file nnet-nnet.h.

References Nnet::AppendComponent(), Nnet::AppendComponentPointer(), Nnet::AppendNnet(), Nnet::components_, Nnet::GetComponent(), Nnet::GetLastComponent(), Nnet::RemoveComponent(), Nnet::RemoveLastComponent(), Nnet::ReplaceComponent(), and Nnet::SwapComponent().

Referenced by Nnet::AppendNnet(), Nnet::Backpropagate(), kaldi::ConvertNnet1ToNnet2(), Nnet::Destroy(), Nnet::Feedforward(), Nnet::GetLastComponent(), Nnet::Info(), Nnet::InfoBackPropagate(), Nnet::InfoGradient(), Nnet::InfoPropagate(), main(), Nnet::Nnet(), Nnet::operator=(), Nnet::Propagate(), Nnet::Read(), Nnet::RemoveLastComponent(), Nnet::ResetStreams(), Nnet::SetDropoutRate(), Nnet::SetSeqLengths(), Nnet::SetTrainOptions(), and Nnet::Write().

                              {
    return components_.size();
  }

NumParams()

int32 NumParams

(

)

const

Get the number of parameters in the network,.

Definition at line 210 of file nnet-nnet.cc.

References Nnet::components_, and rnnlm::n.

Referenced by Nnet::BackpropagateBuffer(), Nnet::GetGradient(), Nnet::GetParams(), Nnet::Info(), MultiBasisComponent::NumParams(), Nnet::SetParams(), and MultiBasisComponent::SetParams().

                            {
  int32 n_params = 0;
  for (int32 n = 0; n < components_.size(); n++) {
    if (components_[n]->IsUpdatable()) {
      n_params +=
        dynamic_cast<UpdatableComponent*>(components_[n])->NumParams();
    }
  }
  return n_params;
}

operator=()

Nnet & operator=

(

const Nnet &

other

)

Definition at line 51 of file nnet-nnet.cc.

References Nnet::backpropagate_buf_, Nnet::Check(), Nnet::components_, Component::Copy(), Nnet::Destroy(), Nnet::GetComponent(), rnnlm::i, Nnet::NumComponents(), Nnet::opts_, Nnet::propagate_buf_, and Nnet::SetTrainOptions().

                                        {
  Destroy();
  // copy the components
  for (int32 i = 0; i < other.NumComponents(); i++) {
    components_.push_back(other.GetComponent(i).Copy());
  }
  // create empty buffers
  propagate_buf_.resize(NumComponents()+1);
  backpropagate_buf_.resize(NumComponents()+1);
  // copy train opts
  SetTrainOptions(other.opts_);
  Check();
  return *this;
}

OutputDim()

int32 OutputDim

(

)

const

Dimensionality of network outputs (posteriors | bn-features | etc.),.

Definition at line 143 of file nnet-nnet.cc.

References Nnet::components_, and KALDI_ASSERT.

Referenced by Nnet::Info(), MultiBasisComponent::InitData(), main(), and MultiBasisComponent::ReadData().

                            {
  KALDI_ASSERT(!components_.empty());
  return components_.back()->OutputDim();
}

Propagate()

void Propagate	(	const CuMatrixBase< BaseFloat > &	in,
		CuMatrix< BaseFloat > *	out
	)

Perform forward pass through the network,.

Forward propagation through the network, (from first component to last).

Definition at line 70 of file nnet-nnet.cc.

References Nnet::components_, rnnlm::i, Nnet::NumComponents(), and Nnet::propagate_buf_.

Referenced by main(), and MultiBasisComponent::PropagateFnc().

                                               {
  // In case of empty network copy input to output,
  if (NumComponents() == 0) {
    (*out) = in;  // copy,
    return;
  }
  // We need C+1 buffers,
  if (propagate_buf_.size() != NumComponents()+1) {
    propagate_buf_.resize(NumComponents()+1);
  }
  // Copy input to first buffer,
  propagate_buf_[0] = in;
  // Propagate through all the components,
  for (int32 i = 0; i < static_cast<int32>(components_.size()); i++) {
    components_[i]->Propagate(propagate_buf_[i], &propagate_buf_[i+1]);
  }
  // Copy the output from the last buffer,
  (*out) = propagate_buf_[NumComponents()];
}

PropagateBuffer()

const std::vector<CuMatrix<BaseFloat> >& PropagateBuffer ( ) const

inline

Access to the forward-pass buffers.

Definition at line 105 of file nnet-nnet.h.

References Nnet::propagate_buf_.

                                                                 {
    return propagate_buf_;
  }

Read() [1/2]

void Read

(

const std::string &

rxfilename

)

Read Nnet from 'rxfilename',.

I/O wrapper for converting 'rxfilename' to 'istream',.

Definition at line 333 of file nnet-nnet.cc.

References Input::Close(), KALDI_WARN, Nnet::NumComponents(), and Input::Stream().

Referenced by Nnet::BackpropagateBuffer(), MultiBasisComponent::InitData(), ParallelComponent::InitData(), Rbm::InitData(), main(), ParallelComponent::ReadData(), and MultiBasisComponent::ReadData().

                                           {
  bool binary;
  Input in(rxfilename, &binary);
  Read(in.Stream(), binary);
  in.Close();
  // Warn if the NN is empty
  if (NumComponents() == 0) {
    KALDI_WARN << "The network '" << rxfilename << "' is empty.";
  }
}

Read() [2/2]

void Read	(	std::istream &	in,
		bool	binary
	)

Read Nnet from 'istream',.

Definition at line 345 of file nnet-nnet.cc.

References Nnet::AppendComponentPointer(), Nnet::Check(), Nnet::components_, Component::InputDim(), KALDI_ERR, Nnet::NumComponents(), and Component::Read().

                                           {
  // Read the Components through the 'factory' Component::Read(...),
  Component* comp(NULL);
  while (comp = Component::Read(is, binary), comp != NULL) {
    // Check dims,
    if (NumComponents() > 0) {
      if (components_.back()->OutputDim() != comp->InputDim()) {
        KALDI_ERR << "Dimensionality mismatch!"
                  << " Previous layer output:" << components_.back()->OutputDim()
                  << " Current layer input:" << comp->InputDim();
      }
    }
    // Append to 'this' Nnet,
    AppendComponentPointer(comp);
  }
  Check();
}

RemoveComponent()

void RemoveComponent

(

int32

c

)

Remove c'th component,.

Definition at line 199 of file nnet-nnet.cc.

References Nnet::Check(), and Nnet::components_.

Referenced by main(), Nnet::NumComponents(), and Nnet::RemoveLastComponent().

                                  {
  Component* ptr = components_.at(c);
  components_.erase(components_.begin()+c);
  delete ptr;
  Check();
}

RemoveLastComponent()

void RemoveLastComponent

(

)

Remove the last of the Components,.

Definition at line 206 of file nnet-nnet.cc.

References Nnet::NumComponents(), and Nnet::RemoveComponent().

Referenced by main(), and Nnet::NumComponents().

                               {
  RemoveComponent(NumComponents()-1);
}

ReplaceComponent()

void ReplaceComponent	(	int32	c,
		const Component &	comp
	)

Replace c'th component in 'this' Nnet (deep copy),.

Definition at line 169 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, and Component::Copy().

Referenced by Nnet::NumComponents().

                                                          {
  delete components_.at(c);
  components_.at(c) = comp.Copy();  // deep copy,
  Check();
}

ResetStreams()

void ResetStreams

(

const std::vector< int32 > &

stream_reset_flag

)

Reset streams in multi-stream training,.

Definition at line 281 of file nnet-nnet.cc.

References Nnet::GetComponent(), Component::IsMultistream(), Nnet::NumComponents(), and MultistreamComponent::ResetStreams().

Referenced by Nnet::BackpropagateBuffer(), and main().

                                                                 {
  for (int32 c = 0; c < NumComponents(); c++) {
    if (GetComponent(c).IsMultistream()) {
      MultistreamComponent& comp =
        dynamic_cast<MultistreamComponent&>(GetComponent(c));
      comp.ResetStreams(stream_reset_flag);
    }
  }
}

SetDropoutRate()

void SetDropoutRate

(

BaseFloat

r

)

Set the dropout rate.

Definition at line 268 of file nnet-nnet.cc.

References Nnet::GetComponent(), Dropout::GetDropoutRate(), Component::GetType(), KALDI_LOG, Component::kDropout, Nnet::NumComponents(), and Dropout::SetDropoutRate().

Referenced by Nnet::BackpropagateBuffer(), and main().

                                      {
  for (int32 c = 0; c < NumComponents(); c++) {
    if (GetComponent(c).GetType() == Component::kDropout) {
      Dropout& comp = dynamic_cast<Dropout&>(GetComponent(c));
      BaseFloat r_old = comp.GetDropoutRate();
      comp.SetDropoutRate(r);
      KALDI_LOG << "Setting dropout-rate in component " << c
                << " from " << r_old << " to " << r;
    }
  }
}

SetParams()

void SetParams

(

const VectorBase< BaseFloat > &

params

)

Set the network weights from a supervector,.

Definition at line 253 of file nnet-nnet.cc.

References Nnet::components_, VectorBase< Real >::Dim(), rnnlm::i, KALDI_ASSERT, Nnet::NumParams(), UpdatableComponent::NumParams(), VectorBase< Real >::Range(), and UpdatableComponent::SetParams().

Referenced by Nnet::BackpropagateBuffer(), and MultiBasisComponent::SetParams().

                                                        {
  KALDI_ASSERT(params.Dim() == NumParams());
  int32 pos = 0;
  // loop over Components,
  for (int32 i = 0; i < components_.size(); i++) {
    if (components_[i]->IsUpdatable()) {
      UpdatableComponent& c =
        dynamic_cast<UpdatableComponent&>(*components_[i]);
      c.SetParams(params.Range(pos, c.NumParams()));  // setting params,
      pos += c.NumParams();
    }
  }
  KALDI_ASSERT(pos == NumParams());
}

SetSeqLengths()

void SetSeqLengths

(

const std::vector< int32 > &

sequence_lengths

)

Set sequence length in LSTM multi-stream training,.

Definition at line 291 of file nnet-nnet.cc.

References Nnet::GetComponent(), Component::IsMultistream(), Nnet::NumComponents(), and MultistreamComponent::SetSeqLengths().

Referenced by Nnet::BackpropagateBuffer(), and main().

                                                                 {
  for (int32 c = 0; c < NumComponents(); c++) {
    if (GetComponent(c).IsMultistream()) {
      MultistreamComponent& comp =
        dynamic_cast<MultistreamComponent&>(GetComponent(c));
      comp.SetSeqLengths(sequence_lengths);
    }
  }
}

SetTrainOptions()

void SetTrainOptions

(

const NnetTrainOptions &

opts

)

Set hyper-parameters of the training (pushes to all UpdatableComponents),.

Definition at line 508 of file nnet-nnet.cc.

References Nnet::GetComponent(), Component::IsUpdatable(), Nnet::NumComponents(), and Nnet::opts_.

Referenced by Nnet::BackpropagateBuffer(), main(), Nnet::Nnet(), Nnet::operator=(), and MultiBasisComponent::SetTrainOptions().

                                                       {
  opts_ = opts;
  // set values to individual components,
  for (int32 l = 0; l < NumComponents(); l++) {
    if (GetComponent(l).IsUpdatable()) {
      dynamic_cast<UpdatableComponent&>(GetComponent(l)).SetTrainOptions(opts_);
    }
  }
}

SwapComponent()

void SwapComponent	(	int32	c,
		Component **	comp
	)

Swap c'th component with the pointer,.

Definition at line 175 of file nnet-nnet.cc.

References Nnet::Check(), and Nnet::components_.

Referenced by Nnet::NumComponents().

                                                  {
  Component* tmp = components_.at(c);
  components_.at(c) = *comp;
  (*comp) = tmp;
  Check();
}

Write() [1/2]

void Write	(	const std::string &	wxfilename,
		bool	binary
	)		const

Write Nnet to 'wxfilename',.

I/O wrapper for converting 'wxfilename' to 'ostream',.

Definition at line 367 of file nnet-nnet.cc.

References Output::Close(), and Output::Stream().

Referenced by Nnet::BackpropagateBuffer(), main(), and MultiBasisComponent::WriteData().

                                                               {
  Output out(wxfilename, binary, true);
  Write(out.Stream(), binary);
  out.Close();
}

Write() [2/2]

void Write	(	std::ostream &	out,
		bool	binary
	)		const

Write Nnet to 'ostream',.

Definition at line 374 of file nnet-nnet.cc.

References Nnet::Check(), Nnet::components_, rnnlm::i, Nnet::NumComponents(), and kaldi::WriteToken().

                                                  {
  Check();
  WriteToken(os, binary, "<Nnet>");
  if (binary == false) os << std::endl;
  for (int32 i = 0; i < NumComponents(); i++) {
    components_[i]->Write(os, binary);
  }
  WriteToken(os, binary, "</Nnet>");
  if (binary == false) os << std::endl;
}

Member Data Documentation

backpropagate_buf_

std::vector<CuMatrix<BaseFloat> > backpropagate_buf_

private

Buffers for backward pass (on demand initialization),.

Definition at line 175 of file nnet-nnet.h.

Referenced by Nnet::Backpropagate(), Nnet::BackpropagateBuffer(), Nnet::Destroy(), Nnet::InfoBackPropagate(), Nnet::Nnet(), and Nnet::operator=().

components_

std::vector<Component*> components_

private

Vector which contains all the components composing the neural network, the components are for example: AffineTransform, Sigmoid, Softmax.

Definition at line 170 of file nnet-nnet.h.

Referenced by Nnet::AppendComponent(), Nnet::AppendComponentPointer(), Nnet::Backpropagate(), Nnet::Check(), Nnet::Destroy(), Nnet::Feedforward(), Nnet::GetComponent(), Nnet::GetGradient(), Nnet::GetLastComponent(), Nnet::GetParams(), Nnet::Info(), Nnet::InfoBackPropagate(), Nnet::InfoGradient(), Nnet::InfoPropagate(), Nnet::InputDim(), Nnet::Nnet(), Nnet::NumComponents(), Nnet::NumParams(), Nnet::operator=(), Nnet::OutputDim(), Nnet::Propagate(), Nnet::Read(), Nnet::RemoveComponent(), Nnet::ReplaceComponent(), Nnet::SetParams(), Nnet::SwapComponent(), and Nnet::Write().

opts_

NnetTrainOptions opts_

private

Option class with hyper-parameters passed to UpdatableComponent(s)

Definition at line 178 of file nnet-nnet.h.

Referenced by Nnet::GetTrainOptions(), Nnet::Nnet(), Nnet::operator=(), and Nnet::SetTrainOptions().

propagate_buf_

std::vector<CuMatrix<BaseFloat> > propagate_buf_

private

Buffers for forward pass (on demand initialization),.

Definition at line 173 of file nnet-nnet.h.

Referenced by Nnet::Backpropagate(), Nnet::Destroy(), Nnet::InfoPropagate(), Nnet::Nnet(), Nnet::operator=(), Nnet::Propagate(), and Nnet::PropagateBuffer().

---

The documentation for this class was generated from the following files:

• nnet/nnet-nnet.h
• nnet/nnet-nnet.cc