Splices a context window of frames together [over time]. More...

#include <nnet-component.h>

Inheritance diagram for SpliceComponent:

Collaboration diagram for SpliceComponent:

Public Member Functions
	SpliceComponent ()

void	Init (int32 input_dim, std::vector< int32 > context, int32 const_component_dim=0)

virtual std::string	Type () const

virtual std::string	Info () const

virtual void	InitFromString (std::string args)
	Initialize, typically from a line of a config file. More...

virtual int32	InputDim () const
	Get size of input vectors. More...

virtual int32	OutputDim () const
	Get size of output vectors. More...

virtual std::vector< int32 >	Context () const
	Return a vector describing the temporal context this component requires for each frame of output, as a sorted list. More...

virtual void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const
	Perform forward pass propagation Input->Output. More...

virtual void	Backprop (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component to_update, CuMatrix< BaseFloat > in_deriv) const
	Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise). More...

virtual bool	BackpropNeedsInput () const

virtual bool	BackpropNeedsOutput () const

virtual Component *	Copy () const
	Copy component (deep copy). More...

virtual void	Read (std::istream &is, bool binary)

virtual void	Write (std::ostream &os, bool binary) const
	Write component to stream. More...

Public Member Functions inherited from Component
	Component ()

virtual int32	Index () const
	Returns the index in the sequence of layers in the neural net; intended only to be used in debugging information. More...

virtual void	SetIndex (int32 index)

void	Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out) const
	A non-virtual propagate function that first resizes output if necessary. More...

virtual	~Component ()

Private Member Functions
	KALDI_DISALLOW_COPY_AND_ASSIGN (SpliceComponent)

Private Attributes
int32	input_dim_

std::vector< int32 >	context_

int32	const_component_dim_

Additional Inherited Members
Static Public Member Functions inherited from Component
static Component *	ReadNew (std::istream &is, bool binary)
	Read component from stream. More...

static Component *	NewFromString (const std::string &initializer_line)
	Initialize the Component from one line that will contain first the type, e.g. More...

static Component *	NewComponentOfType (const std::string &type)
	Return a new Component of the given type e.g. More...

Detailed Description

Splices a context window of frames together [over time].

Definition at line 1092 of file nnet-component.h.

Constructor & Destructor Documentation

◆ SpliceComponent()

SpliceComponent ( )

inline

Definition at line 1094 of file nnet-component.h.

1094 { } // called only prior to Read() or Init().

Member Function Documentation

◆ Backprop()

void Backprop	(	const ChunkInfo &	in_info,
		const ChunkInfo &	out_info,
		const CuMatrixBase< BaseFloat > &	in_value,
		const CuMatrixBase< BaseFloat > &	out_value,
		const CuMatrixBase< BaseFloat > &	out_deriv,
		Component *	to_update,
		CuMatrix< BaseFloat > *	in_deriv
	)		const

virtual

Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise).

Note: in_value and out_value are the values of the input and output of the component, and these may be dummy variables if respectively BackpropNeedsInput() or BackpropNeedsOutput() return false for that component (not all components need these).

num_chunks lets us treat the input matrix as contiguous-in-time chunks of equal size; it only matters if splicing is involved.

Implements Component.

Definition at line 2662 of file nnet-component.cc.

References ChunkInfo::Check(), ChunkInfo::CheckSize(), ChunkInfo::ChunkSize(), CuMatrixBase< Real >::CopyRows(), ChunkInfo::GetIndex(), ChunkInfo::GetOffset(), AffineComponent::InputDim(), KALDI_ASSERT, kaldi::kUndefined, ChunkInfo::NumChunks(), ChunkInfo::NumCols(), CuMatrixBase< Real >::NumCols(), ChunkInfo::NumRows(), CuMatrixBase< Real >::NumRows(), AffineComponent::OutputDim(), and CuMatrix< Real >::Resize().

                                                                     {
   in_info.Check();
   out_info.Check();
   out_info.CheckSize(out_deriv);
   in_deriv->Resize(in_info.NumRows(), in_info.NumCols(), kUndefined);
   KALDI_ASSERT(in_info.NumChunks() == out_info.NumChunks());
   int32 num_chunks = in_info.NumChunks();
   // rewrite backpropagate
 
   int32 out_chunk_size = out_info.ChunkSize(),
          in_chunk_size = in_info.ChunkSize(),
             output_dim = out_deriv.NumCols(),
              input_dim = InputDim();
 
   KALDI_ASSERT(OutputDim() == output_dim);
 
   int32 num_splice = context_.size(),
       const_dim = const_component_dim_;
   // 'indexes' is, for each index from 0 to num_splice - 1,
   // then for each row of "in_deriv", the corresponding row of "out_deriv" that
   // we add, or -1 if.
 
   std::vector<std::vector<int32> > indexes(num_splice);
   // const_dim != 0, "const_indexes" will be used to determine which
   // row of "in" we copy the last part of each row of "out" from (this part is
   // not subject to splicing, it's assumed constant for each frame of "input".
   std::vector<int32> const_indexes(const_dim == 0 ? 0 : in_deriv->NumRows(), -1);
 
   for (int32 c = 0; c < indexes.size(); c++)
     indexes[c].resize(in_deriv->NumRows(), -1);  // set to -1 by default,
   // this gets interpreted by the CopyRows() code
   // as a signal to zero the output...
 
   int32 dim = input_dim - const_dim;  // dimension we are splicing
   for (int32 chunk = 0; chunk < num_chunks; chunk++) {
     if (chunk == 0) { // this branch can be taken for all chunks, but is not
                       // taken for efficiency reasons
       for (int32 c = 0; c < num_splice; c++)  {
         for (int32 out_index = 0; out_index < out_chunk_size; out_index++) {
           int32 out_offset = out_info.GetOffset(out_index);
           int32 in_index = in_info.GetIndex(out_offset + context_[c]);
           indexes[c][chunk * in_chunk_size + in_index] =
               chunk * out_chunk_size + out_index;
         }
       }
     } else {  // just copy the indexes from the previous chunk
       for (int32 c = 0; c < num_splice; c++)  {
         for (int32 in_index = 0; in_index < in_chunk_size; in_index++) {
           int32 last_value = indexes[c][(chunk-1) * in_chunk_size + in_index];
           indexes[c][chunk * in_chunk_size + in_index] =
               (last_value == -1 ? -1 : last_value + out_chunk_size);
         }
       }
     }
     // this code corresponds to the way the forward propagation works; see
     // comments there.
     if (const_dim != 0) {
       for (int32 out_index = 0; out_index < out_chunk_size; out_index++)  {
         const_indexes[chunk * in_chunk_size + out_index] =
             chunk * out_chunk_size + out_index;
       }
     }
   }
 
   CuMatrix<BaseFloat> temp_mat(in_deriv->NumRows(), dim, kUndefined);
 
   for (int32 c = 0; c < num_splice; c++) {
     CuArray<int32> cu_indexes(indexes[c]);
     int32 dim = input_dim - const_dim;  // dimension we
     // are splicing
     CuSubMatrix<BaseFloat> out_deriv_part(out_deriv, 0, out_deriv.NumRows(),
                                           c * dim, dim),
         in_deriv_part(*in_deriv, 0, in_deriv->NumRows(),
                       0, dim);
     if (c == 0) {
       in_deriv_part.CopyRows(out_deriv_part, cu_indexes);
     } else {
       temp_mat.CopyRows(out_deriv_part, cu_indexes);
       in_deriv_part.AddMat(1.0, temp_mat);
     }
   }
   if (const_dim != 0) {
     CuSubMatrix<BaseFloat> out_deriv_part(out_deriv, 0, out_deriv.NumRows(),
                                           out_deriv.NumCols() - const_dim,
                                           const_dim),
         in_deriv_part(*in_deriv, 0, in_deriv->NumRows(),
                       in_deriv->NumCols() - const_dim, const_dim);
     CuArray<int32> cu_const_indexes(const_indexes);
     in_deriv_part.CopyRows(out_deriv_part, cu_const_indexes);
   }
 }

◆ BackpropNeedsInput()

virtual bool BackpropNeedsInput ( ) const

inlinevirtual

Reimplemented from Component.

Definition at line 1119 of file nnet-component.h.

1119 { return false; }

◆ BackpropNeedsOutput()

virtual bool BackpropNeedsOutput ( ) const

inlinevirtual

Reimplemented from Component.

Definition at line 1120 of file nnet-component.h.

References kaldi::cu::Copy(), and KALDI_DISALLOW_COPY_AND_ASSIGN.

1120 { return false; }

◆ Context()

virtual std::vector<int32> Context ( ) const

inlinevirtual

Return a vector describing the temporal context this component requires for each frame of output, as a sorted list.

The default implementation returns a vector ( 0 ), but a splicing layer might return e.g. (-2, -1, 0, 1, 2), but it doesn't have to be contiguous. Note : The context needed by the entire network is a function of the contexts needed by all the components. It is required that Context().front() <= 0 and Context().back() >= 0.

Reimplemented from Component.

Definition at line 1106 of file nnet-component.h.

References Component::Propagate().

1106 { return context_; }

kaldi::nnet2::SpliceComponent::context_

std::vector< int32 > context_

Definition: nnet-component.h:1127

◆ Copy()

Component * Copy ( ) const

virtual

Copy component (deep copy).

Implements Component.

Definition at line 2760 of file nnet-component.cc.

References SpliceComponent::const_component_dim_, SpliceComponent::context_, and SpliceComponent::input_dim_.

                                        {
   SpliceComponent *ans = new SpliceComponent();
   ans->input_dim_ = input_dim_;
   ans->context_ = context_;
   ans->const_component_dim_ = const_component_dim_;
   return ans;
 }

◆ Info()

std::string Info ( ) const

virtual

Reimplemented from Component.

Definition at line 2463 of file nnet-component.cc.

References Component::Info().

                                       {
   std::stringstream stream;
   std::ostringstream os;
   std::copy(context_.begin(), context_.end(),
             std::ostream_iterator<int32>(os, " "));
   stream << Component::Info() << ", context=" << os.str();
   if (const_component_dim_ != 0)
     stream << ", const_component_dim=" << const_component_dim_;
 
   return stream.str();
 }

◆ Init()

void Init	(	int32	input_dim,
		std::vector< int32 >	context,
		int32	const_component_dim = `0`
	)

Definition at line 2475 of file nnet-component.cc.

References kaldi::IsSortedAndUniq(), and KALDI_ASSERT.

Referenced by kaldi::ConvertSpliceComponent(), kaldi::nnet2::GenRandomNnet(), and kaldi::nnet2::UnitTestSpliceComponent().

                                                       {
   input_dim_ = input_dim;
   const_component_dim_ = const_component_dim;
   context_ = context;
   KALDI_ASSERT(context_.size() > 0);
   KALDI_ASSERT(input_dim_ > 0 && context_.front() <= 0 && context_.back() >= 0);
   KALDI_ASSERT(IsSortedAndUniq(context));
   KALDI_ASSERT(const_component_dim_ >= 0 && const_component_dim_ < input_dim_);
 }

◆ InitFromString()

void InitFromString ( std::string args )

virtual

Initialize, typically from a line of a config file.

The "args" will contain any parameters that need to be passed to the Component, e.g. dimensions.

Implements Component.

Definition at line 2488 of file nnet-component.cc.

References rnnlm::i, AffineComponentPreconditionedOnline::Init(), KALDI_ASSERT, KALDI_ERR, kaldi::nnet2::ParseFromString(), and AffineComponentPreconditionedOnline::Type().

                                                    {
   std::string orig_args(args);
   int32 input_dim, left_context, right_context;
   std::vector <int32> context;
   bool in_dim_ok = ParseFromString("input-dim", &args, &input_dim);
   bool context_ok = ParseFromString("context", &args, &context);
   bool left_right_context_ok = ParseFromString("left-context", &args,
                                                &left_context) &&
                                ParseFromString("right-context", &args,
                                                &right_context);
   int32 const_component_dim = 0;
   ParseFromString("const-component-dim", &args, &const_component_dim);
 
   if (!(in_dim_ok && (context_ok || left_right_context_ok)) ||
       !args.empty() || input_dim <= 0)
     KALDI_ERR << "Invalid initializer for layer of type "
               << Type() << ": \"" << orig_args << "\"";
   if (left_right_context_ok)  {
     KALDI_ASSERT(context.size() == 0);
     for (int32 i = -left_context; i <= right_context; i++)
       context.push_back(i);
   }
   Init(input_dim, context, const_component_dim);
 }

◆ InputDim()

virtual int32 InputDim ( ) const

inlinevirtual

Get size of input vectors.

Implements Component.

Definition at line 1104 of file nnet-component.h.

1104 { return input_dim_; }

kaldi::nnet2::SpliceComponent::input_dim_

int32 input_dim_

Definition: nnet-component.h:1126

◆ KALDI_DISALLOW_COPY_AND_ASSIGN()

KALDI_DISALLOW_COPY_AND_ASSIGN ( SpliceComponent )

private

◆ OutputDim()

int32 OutputDim ( ) const

virtual

Get size of output vectors.

Implements Component.

Definition at line 2513 of file nnet-component.cc.

                                        {
   return (input_dim_  - const_component_dim_)
       * (context_.size())
       + const_component_dim_;
 }

◆ Propagate()

void Propagate	(	const ChunkInfo &	in_info,
		const ChunkInfo &	out_info,
		const CuMatrixBase< BaseFloat > &	in,
		CuMatrixBase< BaseFloat > *	out
	)		const

virtual

Perform forward pass propagation Input->Output.

Each row is one frame or training example. Interpreted as "num_chunks" equally sized chunks of frames; this only matters for layers that do things like context splicing. Typically this variable will either be 1 (when we're processing a single contiguous chunk of data) or will be the same as in.NumFrames(), but other values are possible if some layers do splicing.

Implements Component.

Definition at line 2577 of file nnet-component.cc.

References ChunkInfo::Check(), ChunkInfo::CheckSize(), ChunkInfo::ChunkSize(), ChunkInfo::GetIndex(), ChunkInfo::GetOffset(), KALDI_ASSERT, KALDI_ERR, ChunkInfo::NumChunks(), ChunkInfo::NumCols(), CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

                                                                      {
 
   // Check the inputs are correct and resize output
   in_info.Check();
   out_info.Check();
   in_info.CheckSize(in);
   out_info.CheckSize(*out);
   KALDI_ASSERT(in_info.NumChunks() == out_info.NumChunks());
 
   int32 in_chunk_size  = in_info.ChunkSize(),
         out_chunk_size = out_info.ChunkSize(),
         input_dim = in_info.NumCols();
 
   if (out_chunk_size <= 0)
     KALDI_ERR << "Splicing features: output will have zero dimension. "
               << "Probably a code error.";
 
   // 'indexes' is, for each index from 0 to context_.size() - 1,
   // then for each row of "out", the corresponding row of "in" that we copy from
   int32 num_splice = context_.size();
   std::vector<std::vector<int32> > indexes(num_splice);
   for (int32 c = 0; c < num_splice; c++)
     indexes[c].resize(out->NumRows());
   // const_component_dim_ != 0, "const_indexes" will be used to determine which
   // row of "in" we copy the last part of each row of "out" from (this part is
   // not subject to splicing, it's assumed constant for each frame of "input".
   int32 const_dim = const_component_dim_;
   std::vector<int32> const_indexes(const_dim == 0 ? 0 : out->NumRows());
 
   for (int32 chunk = 0; chunk < in_info.NumChunks(); chunk++) {
     if (chunk == 0) {
       // this branch could be used for all chunks in the matrix,
       // but is restricted to chunk 0 for efficiency reasons
       for (int32 c = 0; c < num_splice; c++) {
         for (int32 out_index = 0; out_index < out_chunk_size; out_index++) {
           int32 out_offset = out_info.GetOffset(out_index);
           int32 in_index = in_info.GetIndex(out_offset + context_[c]);
           indexes[c][chunk * out_chunk_size + out_index] =
               chunk * in_chunk_size + in_index;
         }
       }
     } else {  // just copy the indices from the previous chunk
               // and offset these by input chunk size
      for (int32 c = 0; c < num_splice; c++) {
        for (int32 out_index = 0; out_index < out_chunk_size; out_index++) {
          int32 last_value = indexes[c][(chunk-1) * out_chunk_size + out_index];
          indexes[c][chunk * out_chunk_size + out_index] =
              (last_value == -1 ? -1 : last_value + in_chunk_size);
        }
      }
    }
     if (const_dim != 0) {
       for (int32 out_index = 0; out_index < out_chunk_size; out_index++)
         const_indexes[chunk * out_chunk_size + out_index] =
             chunk * in_chunk_size + out_index;  // there is
       // an arbitrariness here; since we assume the const_component
       // is constant within a chunk, it doesn't matter from where we copy.
     }
   }
 
 
   for (int32 c = 0; c < num_splice; c++) {
     int32 dim = input_dim - const_dim;  // dimension we
     // are splicing
     CuSubMatrix<BaseFloat> in_part(in, 0, in.NumRows(),
                                    0, dim),
         out_part(*out, 0, out->NumRows(),
                  c * dim, dim);
     CuArray<int32> cu_indexes(indexes[c]);
     out_part.CopyRows(in_part, cu_indexes);
   }
   if (const_dim != 0) {
     CuSubMatrix<BaseFloat> in_part(in, 0, in.NumRows(),
                                    in.NumCols() - const_dim, const_dim),
         out_part(*out, 0, out->NumRows(),
                  out->NumCols() - const_dim, const_dim);
 
     CuArray<int32> cu_const_indexes(const_indexes);
     out_part.CopyRows(in_part, cu_const_indexes);
   }
 }

◆ Read()

void Read	(	std::istream &	is,
		bool	binary
	)

virtual

Implements Component.

Definition at line 2768 of file nnet-component.cc.

References kaldi::nnet2::ExpectOneOrTwoTokens(), kaldi::ExpectToken(), rnnlm::i, KALDI_ERR, kaldi::ReadBasicType(), kaldi::ReadIntegerVector(), and kaldi::ReadToken().

                                                       {
   ExpectOneOrTwoTokens(is, binary, "<SpliceComponent>", "<InputDim>");
   ReadBasicType(is, binary, &input_dim_);
   std::string token;
   ReadToken(is, false, &token);
   if (token == "<LeftContext>") {
     int32 left_context=0, right_context=0;
     std::vector<int32> context;
     ReadBasicType(is, binary, &left_context);
     ExpectToken(is, binary, "<RightContext>");
     ReadBasicType(is, binary, &right_context);
     for (int32 i = -1 * left_context; i <= right_context; i++)
       context.push_back(i);
     context_ = context;
   } else  if (token == "<Context>") {
     ReadIntegerVector(is, binary, &context_);
   } else  {
     KALDI_ERR << "Unknown token" << token
               << ", the model might be corrupted";
   }
   ExpectToken(is, binary, "<ConstComponentDim>");
   ReadBasicType(is, binary, &const_component_dim_);
   ExpectToken(is, binary, "</SpliceComponent>");
 }

◆ Type()

virtual std::string Type ( ) const

inlinevirtual

Implements Component.

Definition at line 1101 of file nnet-component.h.

1101 { return "SpliceComponent"; }

◆ Write()

void Write	(	std::ostream &	os,
		bool	binary
	)		const

virtual

Write component to stream.

Implements Component.

Definition at line 2793 of file nnet-component.cc.

References kaldi::WriteBasicType(), kaldi::WriteIntegerVector(), and kaldi::WriteToken().

                                                              {
   WriteToken(os, binary, "<SpliceComponent>");
   WriteToken(os, binary, "<InputDim>");
   WriteBasicType(os, binary, input_dim_);
   WriteToken(os, binary, "<Context>");
   WriteIntegerVector(os, binary, context_);
   WriteToken(os, binary, "<ConstComponentDim>");
   WriteBasicType(os, binary, const_component_dim_);
   WriteToken(os, binary, "</SpliceComponent>");
 }

Member Data Documentation

◆ const_component_dim_

int32 const_component_dim_

private

Definition at line 1128 of file nnet-component.h.

Referenced by SpliceComponent::Copy().

◆ context_

std::vector<int32> context_

private

Definition at line 1127 of file nnet-component.h.

Referenced by SpliceComponent::Copy().

◆ input_dim_

int32 input_dim_

private

Definition at line 1126 of file nnet-component.h.

Referenced by SpliceComponent::Copy().

The documentation for this class was generated from the following files:

nnet2/nnet-component.h
nnet2/nnet-component.cc

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ SpliceComponent()

Member Function Documentation

◆ Backprop()

◆ BackpropNeedsInput()

◆ BackpropNeedsOutput()

◆ Context()

◆ Copy()

◆ Info()

◆ Init()

◆ InitFromString()

◆ InputDim()

◆ KALDI_DISALLOW_COPY_AND_ASSIGN()

◆ OutputDim()

◆ Propagate()

◆ Read()

◆ Type()

◆ Write()

Member Data Documentation

◆ const_component_dim_

◆ context_

◆ input_dim_