All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
BlockAffineComponent Class Reference

#include <nnet-component.h>

Inheritance diagram for BlockAffineComponent:
Collaboration diagram for BlockAffineComponent:

Public Member Functions

virtual int32 InputDim () const
 Get size of input vectors. More...
 
virtual int32 OutputDim () const
 Get size of output vectors. More...
 
virtual int32 GetParameterDim () const
 The following new virtual function returns the total dimension of the parameters in this class. More...
 
virtual void Vectorize (VectorBase< BaseFloat > *params) const
 Turns the parameters into vector form. More...
 
virtual void UnVectorize (const VectorBase< BaseFloat > &params)
 Converts the parameters from vector form. More...
 
void Init (BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 num_blocks)
 
virtual void InitFromString (std::string args)
 Initialize, typically from a line of a config file. More...
 
 BlockAffineComponent ()
 
virtual std::string Type () const
 
virtual bool BackpropNeedsInput () const
 
virtual bool BackpropNeedsOutput () const
 
virtual void Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const
 Perform forward pass propagation Input->Output. More...
 
virtual void Backprop (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component *to_update, CuMatrix< BaseFloat > *in_deriv) const
 Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise). More...
 
virtual void SetZero (bool treat_as_gradient)
 Set parameters to zero, and if treat_as_gradient is true, we'll be treating this as a gradient so set the learning rate to 1 and make any other changes necessary (there's a variable we have to set for the MixtureProbComponent). More...
 
virtual void Read (std::istream &is, bool binary)
 
virtual void Write (std::ostream &os, bool binary) const
 Write component to stream. More...
 
virtual BaseFloat DotProduct (const UpdatableComponent &other) const
 Here, "other" is a component of the same specific type. More...
 
virtual ComponentCopy () const
 Copy component (deep copy). More...
 
virtual void PerturbParams (BaseFloat stddev)
 We introduce a new virtual function that only applies to class UpdatableComponent. More...
 
virtual void Scale (BaseFloat scale)
 This new virtual function scales the parameters by this amount. More...
 
virtual void Add (BaseFloat alpha, const UpdatableComponent &other)
 This new virtual function adds the parameters of another updatable component, times some constant, to the current parameters. More...
 
- Public Member Functions inherited from UpdatableComponent
 UpdatableComponent (const UpdatableComponent &other)
 
void Init (BaseFloat learning_rate)
 
 UpdatableComponent (BaseFloat learning_rate)
 
 UpdatableComponent ()
 
virtual ~UpdatableComponent ()
 
void SetLearningRate (BaseFloat lrate)
 Sets the learning rate of gradient descent. More...
 
BaseFloat LearningRate () const
 Gets the learning rate of gradient descent. More...
 
virtual std::string Info () const
 
- Public Member Functions inherited from Component
 Component ()
 
virtual int32 Index () const
 Returns the index in the sequence of layers in the neural net; intended only to be used in debugging information. More...
 
virtual void SetIndex (int32 index)
 
virtual std::vector< int32 > Context () const
 Return a vector describing the temporal context this component requires for each frame of output, as a sorted list. More...
 
void Propagate (const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out) const
 A non-virtual propagate function that first resizes output if necessary. More...
 
virtual ~Component ()
 

Protected Member Functions

virtual void Update (const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_deriv)
 
virtual void UpdateSimple (const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_deriv)
 

Protected Attributes

CuMatrix< BaseFloatlinear_params_
 
CuVector< BaseFloatbias_params_
 
int32 num_blocks_
 
- Protected Attributes inherited from UpdatableComponent
BaseFloat learning_rate_
 learning rate (0.0..0.01) More...
 

Private Member Functions

 KALDI_DISALLOW_COPY_AND_ASSIGN (BlockAffineComponent)
 

Additional Inherited Members

- Static Public Member Functions inherited from Component
static ComponentReadNew (std::istream &is, bool binary)
 Read component from stream. More...
 
static ComponentNewFromString (const std::string &initializer_line)
 Initialize the Component from one line that will contain first the type, e.g. More...
 
static ComponentNewComponentOfType (const std::string &type)
 Return a new Component of the given type e.g. More...
 

Detailed Description

Definition at line 1171 of file nnet-component.h.

Constructor & Destructor Documentation

Definition at line 1186 of file nnet-component.h.

Referenced by BlockAffineComponent::Copy().

1186 { } // use Init to really initialize.

Member Function Documentation

void Add ( BaseFloat  alpha,
const UpdatableComponent other 
)
virtual

This new virtual function adds the parameters of another updatable component, times some constant, to the current parameters.

Implements UpdatableComponent.

Definition at line 1965 of file nnet-component.cc.

References CuMatrixBase< Real >::AddMat(), CuVectorBase< Real >::AddVec(), BlockAffineComponent::bias_params_, KALDI_ASSERT, and BlockAffineComponent::linear_params_.

1966  {
1967  const BlockAffineComponent *other =
1968  dynamic_cast<const BlockAffineComponent*>(&other_in);
1969  KALDI_ASSERT(other != NULL);
1970  linear_params_.AddMat(alpha, other->linear_params_);
1971  bias_params_.AddVec(alpha, other->bias_params_);
1972 }
void AddMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType trans=kNoTrans)
*this += alpha * A
Definition: cu-matrix.cc:939
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void AddVec(Real alpha, const CuVectorBase< Real > &vec, Real beta=1.0)
Definition: cu-vector.cc:1126
void Backprop ( const ChunkInfo in_info,
const ChunkInfo out_info,
const CuMatrixBase< BaseFloat > &  in_value,
const CuMatrixBase< BaseFloat > &  out_value,
const CuMatrixBase< BaseFloat > &  out_deriv,
Component to_update,
CuMatrix< BaseFloat > *  in_deriv 
) const
virtual

Perform backward pass propagation of the derivative, and also either update the model (if to_update == this) or update another model or compute the model derivative (otherwise).

Note: in_value and out_value are the values of the input and output of the component, and these may be dummy variables if respectively BackpropNeedsInput() or BackpropNeedsOutput() return false for that component (not all components need these).

num_chunks lets us treat the input matrix as contiguous-in-time chunks of equal size; it only matters if splicing is involved.

Implements Component.

Definition at line 2035 of file nnet-component.cc.

References BlockAffineComponent::InputDim(), KALDI_ASSERT, kaldi::kNoTrans, BlockAffineComponent::linear_params_, BlockAffineComponent::num_blocks_, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), CuMatrix< Real >::Resize(), and BlockAffineComponent::Update().

2041  {
2042 
2043  // This code mirrors the code in Propagate().
2044  int32 num_frames = in_value.NumRows();
2045  BlockAffineComponent *to_update = dynamic_cast<BlockAffineComponent*>(
2046  to_update_in);
2047  in_deriv->Resize(out_deriv.NumRows(), InputDim());
2048  int32 input_block_dim = linear_params_.NumCols(),
2049  output_block_dim = linear_params_.NumRows() / num_blocks_;
2050  KALDI_ASSERT(in_value.NumCols() == input_block_dim * num_blocks_);
2051  KALDI_ASSERT(out_deriv.NumCols() == output_block_dim * num_blocks_);
2052 
2053  for (int32 b = 0; b < num_blocks_; b++) {
2054  CuSubMatrix<BaseFloat> in_value_block(in_value, 0, num_frames,
2055  b * input_block_dim,
2056  input_block_dim),
2057  in_deriv_block(*in_deriv, 0, num_frames,
2058  b * input_block_dim, input_block_dim),
2059  out_deriv_block(out_deriv, 0, num_frames,
2060  b * output_block_dim, output_block_dim),
2061  param_block(linear_params_,
2062  b * output_block_dim, output_block_dim,
2063  0, input_block_dim);
2064 
2065  // Propagate the derivative back to the input.
2066  in_deriv_block.AddMatMat(1.0, out_deriv_block, kNoTrans,
2067  param_block, kNoTrans, 0.0);
2068  }
2069  if (to_update != NULL)
2070  to_update->Update(in_value, out_deriv);
2071 }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
virtual int32 InputDim() const
Get size of input vectors.
CuMatrix< BaseFloat > linear_params_
virtual bool BackpropNeedsInput ( ) const
inlinevirtual

Reimplemented from Component.

Definition at line 1188 of file nnet-component.h.

1188 { return true; }
virtual bool BackpropNeedsOutput ( ) const
inlinevirtual

Reimplemented from Component.

Definition at line 1189 of file nnet-component.h.

1189 { return false; }
Component * Copy ( ) const
virtual

Copy component (deep copy).

Implements Component.

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 1951 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, BlockAffineComponent::BlockAffineComponent(), UpdatableComponent::learning_rate_, BlockAffineComponent::linear_params_, and BlockAffineComponent::num_blocks_.

1951  {
1953  ans->learning_rate_ = learning_rate_;
1954  ans->linear_params_ = linear_params_;
1955  ans->bias_params_ = bias_params_;
1956  ans->num_blocks_ = num_blocks_;
1957  return ans;
1958 }
BaseFloat learning_rate_
learning rate (0.0..0.01)
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
BaseFloat DotProduct ( const UpdatableComponent other) const
virtual

Here, "other" is a component of the same specific type.

This function computes the dot product in parameters, and is computed while automatically adjusting learning rates; typically, one of the two will actually contain the gradient.

Implements UpdatableComponent.

Definition at line 1943 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, kaldi::kTrans, BlockAffineComponent::linear_params_, kaldi::TraceMatMat(), and kaldi::VecVec().

1944  {
1945  const BlockAffineComponent *other =
1946  dynamic_cast<const BlockAffineComponent*>(&other_in);
1947  return TraceMatMat(linear_params_, other->linear_params_, kTrans)
1948  + VecVec(bias_params_, other->bias_params_);
1949 }
Real TraceMatMat(const MatrixBase< Real > &A, const MatrixBase< Real > &B, MatrixTransposeType trans)
We need to declare this here as it will be a friend function.
CuVector< BaseFloat > bias_params_
Real VecVec(const VectorBase< Real > &a, const VectorBase< Real > &b)
Returns dot product between v1 and v2.
Definition: kaldi-vector.cc:36
CuMatrix< BaseFloat > linear_params_
int32 GetParameterDim ( ) const
virtual

The following new virtual function returns the total dimension of the parameters in this class.

E.g. used for L-BFGS update

Reimplemented from UpdatableComponent.

Definition at line 2142 of file nnet-component.cc.

References BlockAffineComponent::InputDim(), BlockAffineComponent::num_blocks_, and BlockAffineComponent::OutputDim().

2142  {
2143  // Note: num_blocks_ should divide both InputDim() and OutputDim().
2144  return InputDim() * OutputDim() / num_blocks_;
2145 }
virtual int32 OutputDim() const
Get size of output vectors.
virtual int32 InputDim() const
Get size of input vectors.
void Init ( BaseFloat  learning_rate,
int32  input_dim,
int32  output_dim,
BaseFloat  param_stddev,
BaseFloat  bias_stddev,
int32  num_blocks 
)

Definition at line 2074 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, UpdatableComponent::Init(), KALDI_ASSERT, BlockAffineComponent::linear_params_, BlockAffineComponent::num_blocks_, CuVector< Real >::Resize(), CuMatrix< Real >::Resize(), CuVectorBase< Real >::Scale(), CuMatrixBase< Real >::Scale(), CuVectorBase< Real >::SetRandn(), and CuMatrixBase< Real >::SetRandn().

Referenced by BlockAffineComponentPreconditioned::Init(), BlockAffineComponent::InitFromString(), and kaldi::nnet2::UnitTestBlockAffineComponent().

2078  {
2079  UpdatableComponent::Init(learning_rate);
2080  KALDI_ASSERT(output_dim > 0 && input_dim > 0 && param_stddev >= 0.0);
2081  KALDI_ASSERT(input_dim % num_blocks == 0 && output_dim % num_blocks == 0);
2082 
2083  linear_params_.Resize(output_dim, input_dim / num_blocks);
2084  bias_params_.Resize(output_dim);
2085 
2086  linear_params_.SetRandn(); // sets to random normally distributed noise.
2087  linear_params_.Scale(param_stddev);
2089  bias_params_.Scale(bias_stddev);
2090  num_blocks_ = num_blocks;
2091 }
void Scale(Real value)
Definition: cu-vector.cc:1105
void Scale(Real value)
Definition: cu-matrix.cc:608
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
void Init(BaseFloat learning_rate)
void Resize(MatrixIndexT rows, MatrixIndexT cols, MatrixResizeType resize_type=kSetZero, MatrixStrideType stride_type=kDefaultStride)
Allocate the memory.
Definition: cu-matrix.cc:47
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void InitFromString ( std::string  args)
virtual

Initialize, typically from a line of a config file.

The "args" will contain any parameters that need to be passed to the Component, e.g. dimensions.

Implements Component.

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 2093 of file nnet-component.cc.

References BlockAffineComponent::Init(), KALDI_ERR, UpdatableComponent::learning_rate_, and kaldi::nnet2::ParseFromString().

Referenced by kaldi::nnet2::UnitTestBlockAffineComponent().

2093  {
2094  std::string orig_args(args);
2095  bool ok = true;
2096  BaseFloat learning_rate = learning_rate_;
2097  int32 input_dim = -1, output_dim = -1, num_blocks = 1;
2098  ParseFromString("learning-rate", &args, &learning_rate); // optional.
2099  ok = ok && ParseFromString("input-dim", &args, &input_dim);
2100  ok = ok && ParseFromString("output-dim", &args, &output_dim);
2101  ok = ok && ParseFromString("num-blocks", &args, &num_blocks);
2102  BaseFloat param_stddev = 1.0 / std::sqrt(input_dim),
2103  bias_stddev = 1.0;
2104  ParseFromString("param-stddev", &args, &param_stddev);
2105  ParseFromString("bias-stddev", &args, &bias_stddev);
2106  if (!args.empty())
2107  KALDI_ERR << "Could not process these elements in initializer: "
2108  << args;
2109  if (!ok)
2110  KALDI_ERR << "Bad initializer " << orig_args;
2111  Init(learning_rate, input_dim, output_dim,
2112  param_stddev, bias_stddev, num_blocks);
2113 }
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 num_blocks)
bool ParseFromString(const std::string &name, std::string *string, int32 *param)
Functions used in Init routines.
float BaseFloat
Definition: kaldi-types.h:29
BaseFloat learning_rate_
learning rate (0.0..0.01)
#define KALDI_ERR
Definition: kaldi-error.h:127
virtual int32 InputDim ( ) const
inlinevirtual
KALDI_DISALLOW_COPY_AND_ASSIGN ( BlockAffineComponent  )
private
virtual int32 OutputDim ( ) const
inlinevirtual

Get size of output vectors.

Implements Component.

Definition at line 1174 of file nnet-component.h.

References BlockAffineComponent::linear_params_, and CuMatrixBase< Real >::NumRows().

Referenced by BlockAffineComponent::GetParameterDim().

1174 { return linear_params_.NumRows(); }
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuMatrix< BaseFloat > linear_params_
void PerturbParams ( BaseFloat  stddev)
virtual

We introduce a new virtual function that only applies to class UpdatableComponent.

This is used in testing.

Implements UpdatableComponent.

Definition at line 1933 of file nnet-component.cc.

References CuMatrixBase< Real >::AddMat(), CuVectorBase< Real >::AddVec(), BlockAffineComponent::bias_params_, BlockAffineComponent::linear_params_, CuVectorBase< Real >::SetRandn(), and CuMatrixBase< Real >::SetRandn().

1933  {
1934  CuMatrix<BaseFloat> temp_linear_params(linear_params_);
1935  temp_linear_params.SetRandn();
1936  linear_params_.AddMat(stddev, temp_linear_params);
1937 
1938  CuVector<BaseFloat> temp_bias_params(bias_params_);
1939  temp_bias_params.SetRandn();
1940  bias_params_.AddVec(stddev, temp_bias_params);
1941 }
void AddMat(Real alpha, const CuMatrixBase< Real > &A, MatrixTransposeType trans=kNoTrans)
*this += alpha * A
Definition: cu-matrix.cc:939
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void AddVec(Real alpha, const CuVectorBase< Real > &vec, Real beta=1.0)
Definition: cu-vector.cc:1126
void Propagate ( const ChunkInfo in_info,
const ChunkInfo out_info,
const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
) const
virtual

Perform forward pass propagation Input->Output.

Each row is one frame or training example. Interpreted as "num_chunks" equally sized chunks of frames; this only matters for layers that do things like context splicing. Typically this variable will either be 1 (when we're processing a single contiguous chunk of data) or will be the same as in.NumFrames(), but other values are possible if some layers do splicing.

Implements Component.

Definition at line 1974 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, ChunkInfo::CheckSize(), CuMatrixBase< Real >::CopyRowsFromVec(), KALDI_ASSERT, kaldi::kNoTrans, kaldi::kTrans, BlockAffineComponent::linear_params_, BlockAffineComponent::num_blocks_, ChunkInfo::NumChunks(), CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

1977  {
1978  in_info.CheckSize(in);
1979  out_info.CheckSize(*out);
1980  KALDI_ASSERT(in_info.NumChunks() == out_info.NumChunks());
1981 
1982  // The matrix has a block structure where each matrix has input dim
1983  // (#rows) equal to input_block_dim. The blocks are stored in linear_params_
1984  // as [ M
1985  // N
1986  // O ] but we actually treat it as:
1987  // [ M 0 0
1988  // 0 N 0
1989  // 0 0 O ]
1990  int32 input_block_dim = linear_params_.NumCols(),
1991  output_block_dim = linear_params_.NumRows() / num_blocks_,
1992  num_frames = in.NumRows();
1993  KALDI_ASSERT(in.NumCols() == input_block_dim * num_blocks_);
1994  KALDI_ASSERT(out->NumCols() == output_block_dim * num_blocks_);
1995  KALDI_ASSERT(in.NumRows() == out->NumRows());
1996 
1997  out->CopyRowsFromVec(bias_params_); // copies bias_params_ to each row
1998  // of *out.
1999 
2000  for (int32 b = 0; b < num_blocks_; b++) {
2001  CuSubMatrix<BaseFloat> in_block(in, 0, num_frames,
2002  b * input_block_dim, input_block_dim),
2003  out_block(*out, 0, num_frames,
2004  b * output_block_dim, output_block_dim),
2005  param_block(linear_params_,
2006  b * output_block_dim, output_block_dim,
2007  0, input_block_dim);
2008  out_block.AddMatMat(1.0, in_block, kNoTrans, param_block, kTrans, 1.0);
2009  }
2010 }
void CopyRowsFromVec(const CuVectorBase< Real > &v)
This function has two modes of operation.
Definition: cu-matrix.cc:2144
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void Read ( std::istream &  is,
bool  binary 
)
virtual

Implements Component.

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 2116 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, kaldi::nnet2::ExpectOneOrTwoTokens(), kaldi::ExpectToken(), UpdatableComponent::learning_rate_, BlockAffineComponent::linear_params_, BlockAffineComponent::num_blocks_, CuVector< Real >::Read(), CuMatrix< Real >::Read(), and kaldi::ReadBasicType().

2116  {
2117  ExpectOneOrTwoTokens(is, binary, "<BlockAffineComponent>", "<LearningRate>");
2118  ReadBasicType(is, binary, &learning_rate_);
2119  ExpectToken(is, binary, "<NumBlocks>");
2120  ReadBasicType(is, binary, &num_blocks_);
2121  ExpectToken(is, binary, "<LinearParams>");
2122  linear_params_.Read(is, binary);
2123  ExpectToken(is, binary, "<BiasParams>");
2124  bias_params_.Read(is, binary);
2125  ExpectToken(is, binary, "</BlockAffineComponent>");
2126 }
void ReadBasicType(std::istream &is, bool binary, T *t)
ReadBasicType is the name of the read function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:55
void Read(std::istream &is, bool binary)
I/O.
Definition: cu-vector.cc:862
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
BaseFloat learning_rate_
learning rate (0.0..0.01)
void Read(std::istream &is, bool binary)
I/O functions.
Definition: cu-matrix.cc:459
static void ExpectOneOrTwoTokens(std::istream &is, bool binary, const std::string &token1, const std::string &token2)
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void Scale ( BaseFloat  scale)
virtual

This new virtual function scales the parameters by this amount.

Implements UpdatableComponent.

Definition at line 1960 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, BlockAffineComponent::linear_params_, CuVectorBase< Real >::Scale(), and CuMatrixBase< Real >::Scale().

1960  {
1961  linear_params_.Scale(scale);
1962  bias_params_.Scale(scale);
1963 }
void Scale(Real value)
Definition: cu-vector.cc:1105
void Scale(Real value)
Definition: cu-matrix.cc:608
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void SetZero ( bool  treat_as_gradient)
virtual

Set parameters to zero, and if treat_as_gradient is true, we'll be treating this as a gradient so set the learning rate to 1 and make any other changes necessary (there's a variable we have to set for the MixtureProbComponent).

Implements UpdatableComponent.

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 1925 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, BlockAffineComponent::linear_params_, UpdatableComponent::SetLearningRate(), CuVectorBase< Real >::SetZero(), and CuMatrixBase< Real >::SetZero().

Referenced by BlockAffineComponentPreconditioned::SetZero().

1925  {
1926  if (treat_as_gradient) {
1927  SetLearningRate(1.0);
1928  }
1931 }
void SetZero()
Math operations, some calling kernels.
Definition: cu-matrix.cc:474
void SetZero()
Math operations.
Definition: cu-vector.cc:988
void SetLearningRate(BaseFloat lrate)
Sets the learning rate of gradient descent.
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
virtual std::string Type ( ) const
inlinevirtual

Implements Component.

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 1187 of file nnet-component.h.

1187 { return "BlockAffineComponent"; }
void UnVectorize ( const VectorBase< BaseFloat > &  params)
virtual

Converts the parameters from vector form.

Reimplemented from UpdatableComponent.

Definition at line 2153 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, CuVectorBase< Real >::CopyFromVec(), CuMatrixBase< Real >::CopyRowsFromVec(), CuVectorBase< Real >::Dim(), BlockAffineComponent::linear_params_, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), and VectorBase< Real >::Range().

2153  {
2154  int32 l = linear_params_.NumRows() * linear_params_.NumCols(),
2155  b = bias_params_.Dim();
2156  linear_params_.CopyRowsFromVec(params.Range(0, l));
2157  bias_params_.CopyFromVec(params.Range(l, b));
2158 }
void CopyRowsFromVec(const CuVectorBase< Real > &v)
This function has two modes of operation.
Definition: cu-matrix.cc:2144
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void CopyFromVec(const CuVectorBase< Real > &src)
Copy functions; these will crash if the dimension do not match.
Definition: cu-vector.cc:970
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
virtual void Update ( const CuMatrixBase< BaseFloat > &  in_value,
const CuMatrixBase< BaseFloat > &  out_deriv 
)
inlineprotectedvirtual

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 1211 of file nnet-component.h.

References BlockAffineComponent::UpdateSimple().

Referenced by BlockAffineComponent::Backprop().

1213  {
1214  UpdateSimple(in_value, out_deriv);
1215  }
virtual void UpdateSimple(const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_deriv)
void UpdateSimple ( const CuMatrixBase< BaseFloat > &  in_value,
const CuMatrixBase< BaseFloat > &  out_deriv 
)
protectedvirtual

Definition at line 2012 of file nnet-component.cc.

References CuMatrixBase< Real >::AddMatMat(), CuVectorBase< Real >::AddRowSumMat(), BlockAffineComponent::bias_params_, kaldi::kNoTrans, kaldi::kTrans, UpdatableComponent::learning_rate_, BlockAffineComponent::linear_params_, BlockAffineComponent::num_blocks_, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

Referenced by BlockAffineComponent::Update(), and BlockAffineComponentPreconditioned::Update().

2014  {
2015  int32 input_block_dim = linear_params_.NumCols(),
2016  output_block_dim = linear_params_.NumRows() / num_blocks_,
2017  num_frames = in_value.NumRows();
2018 
2019  bias_params_.AddRowSumMat(learning_rate_, out_deriv, 1.0);
2020  for (int32 b = 0; b < num_blocks_; b++) {
2021  CuSubMatrix<BaseFloat> in_value_block(in_value, 0, num_frames,
2022  b * input_block_dim,
2023  input_block_dim),
2024  out_deriv_block(out_deriv, 0, num_frames,
2025  b * output_block_dim, output_block_dim),
2026  param_block(linear_params_,
2027  b * output_block_dim, output_block_dim,
2028  0, input_block_dim);
2029  // Update the parameters.
2030  param_block.AddMatMat(learning_rate_, out_deriv_block, kTrans,
2031  in_value_block, kNoTrans, 1.0);
2032  }
2033 }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void AddRowSumMat(Real alpha, const CuMatrixBase< Real > &mat, Real beta=1.0)
Sum the rows of the matrix, add to vector.
Definition: cu-vector.cc:1166
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
BaseFloat learning_rate_
learning rate (0.0..0.01)
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void Vectorize ( VectorBase< BaseFloat > *  params) const
virtual

Turns the parameters into vector form.

We put the vector form on the CPU, because in the kinds of situations where we do this, we'll tend to use too much memory for the GPU.

Reimplemented from UpdatableComponent.

Definition at line 2147 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, CuVectorBase< Real >::Dim(), BlockAffineComponent::linear_params_, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), and VectorBase< Real >::Range().

2147  {
2148  int32 l = linear_params_.NumRows() * linear_params_.NumCols(),
2149  b = bias_params_.Dim();
2150  params->Range(0, l).CopyRowsFromMat(linear_params_);
2151  params->Range(l, b).CopyFromVec(bias_params_);
2152 }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
MatrixIndexT Dim() const
Dimensions.
Definition: cu-vector.h:67
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
SubVector< Real > Range(const MatrixIndexT o, const MatrixIndexT l)
Returns a sub-vector of a vector (a range of elements).
Definition: kaldi-vector.h:92
void Write ( std::ostream &  os,
bool  binary 
) const
virtual

Write component to stream.

Implements Component.

Reimplemented in BlockAffineComponentPreconditioned.

Definition at line 2128 of file nnet-component.cc.

References BlockAffineComponent::bias_params_, UpdatableComponent::learning_rate_, BlockAffineComponent::linear_params_, BlockAffineComponent::num_blocks_, CuVector< Real >::Write(), CuMatrixBase< Real >::Write(), kaldi::WriteBasicType(), and kaldi::WriteToken().

2128  {
2129  WriteToken(os, binary, "<BlockAffineComponent>");
2130  WriteToken(os, binary, "<LearningRate>");
2131  WriteBasicType(os, binary, learning_rate_);
2132  WriteToken(os, binary, "<NumBlocks>");
2133  WriteBasicType(os, binary, num_blocks_);
2134  WriteToken(os, binary, "<LinearParams>");
2135  linear_params_.Write(os, binary);
2136  WriteToken(os, binary, "<BiasParams>");
2137  bias_params_.Write(os, binary);
2138  WriteToken(os, binary, "</BlockAffineComponent>");
2139 }
void Write(std::ostream &is, bool binary) const
Definition: cu-vector.cc:872
BaseFloat learning_rate_
learning rate (0.0..0.01)
void WriteToken(std::ostream &os, bool binary, const char *token)
The WriteToken functions are for writing nonempty sequences of non-space characters.
Definition: io-funcs.cc:134
void WriteBasicType(std::ostream &os, bool binary, T t)
WriteBasicType is the name of the write function for bool, integer types, and floating-point types...
Definition: io-funcs-inl.h:34
CuVector< BaseFloat > bias_params_
CuMatrix< BaseFloat > linear_params_
void Write(std::ostream &os, bool binary) const
Definition: cu-matrix.cc:467

Member Data Documentation


The documentation for this class was generated from the following files: