AffineTransform Class Reference

#include <nnet-affine-transform.h>

Inheritance diagram for AffineTransform:

Collaboration diagram for AffineTransform:

Public Member Functions
	AffineTransform (int32 dim_in, int32 dim_out)
	~AffineTransform ()
Component *	Copy () const
	Copy component (deep copy),. More...
ComponentType	GetType () const
	Get Type Identification of the component,. More...
void	InitData (std::istream &is)
	Initialize the content of the component by the 'line' from the prototype,. More...
void	ReadData (std::istream &is, bool binary)
	Reads the component content. More...
void	WriteData (std::ostream &os, bool binary) const
	Writes the component content. More...
int32	NumParams () const
	Number of trainable parameters,. More...
void	GetGradient (VectorBase< BaseFloat > *gradient) const
	Get gradient reshaped as a vector,. More...
void	GetParams (VectorBase< BaseFloat > *params) const
	Get the trainable parameters reshaped as a vector,. More...
void	SetParams (const VectorBase< BaseFloat > &params)
	Set the trainable parameters from, reshaped as a vector,. More...
std::string	Info () const
	Print some additional info (after <ComponentName> and the dims),. More...
std::string	InfoGradient () const
	Print some additional info about gradient (after <...> and dims),. More...
void	PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
	Abstract interface for propagation/backpropagation. More...
void	BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
	Backward pass transformation (to be implemented by descending class...) More...
void	Update (const CuMatrixBase< BaseFloat > &input, const CuMatrixBase< BaseFloat > &diff)
	Compute gradient and update parameters,. More...
const CuVectorBase< BaseFloat > &	GetBias () const
	Accessors to the component parameters,. More...
void	SetBias (const CuVectorBase< BaseFloat > &bias)
const CuMatrixBase< BaseFloat > &	GetLinearity () const
void	SetLinearity (const CuMatrixBase< BaseFloat > &linearity)
Public Member Functions inherited from UpdatableComponent
	UpdatableComponent (int32 input_dim, int32 output_dim)
virtual	~UpdatableComponent ()
bool	IsUpdatable () const
	Check if contains trainable parameters,. More...
virtual void	SetTrainOptions (const NnetTrainOptions &opts)
	Set the training options to the component,. More...
const NnetTrainOptions &	GetTrainOptions () const
	Get the training options from the component,. More...
virtual void	SetLearnRateCoef (BaseFloat val)
	Set the learn-rate coefficient,. More...
virtual void	SetBiasLearnRateCoef (BaseFloat val)
	Set the learn-rate coefficient for bias,. More...
Public Member Functions inherited from Component
	Component (int32 input_dim, int32 output_dim)
	Generic interface of a component,. More...
virtual	~Component ()
virtual bool	IsMultistream () const
	Check if component has 'Recurrent' interface (trainable and recurrent),. More...
int32	InputDim () const
	Get the dimension of the input,. More...
int32	OutputDim () const
	Get the dimension of the output,. More...
void	Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward-pass propagation 'in' -> 'out',. More...
void	Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
	Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
void	Write (std::ostream &os, bool binary) const
	Write the component to a stream,. More...

Private Attributes
CuMatrix< BaseFloat >	linearity_
CuVector< BaseFloat >	bias_
CuMatrix< BaseFloat >	linearity_corr_
CuVector< BaseFloat >	bias_corr_
BaseFloat	max_norm_

Additional Inherited Members
Public Types inherited from Component
enum	ComponentType {   kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,   kConvolutionalComponent, kLstmProjected, kBlstmProjected, kRecurrentComponent,   kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax, kBlockSoftmax,   kSigmoid, kTanh, kParametricRelu, kDropout,   kLengthNormComponent, kTranform = 0x0400, kRbm, kSplice,   kCopy, kTranspose, kBlockLinearity, kAddShift,   kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent, kSimpleSentenceAveragingComponent,   kAveragePoolingComponent, kMaxPoolingComponent, kFramePoolingComponent, kParallelComponent,   kMultiBasisComponent }
	Component type identification mechanism,. More...
Static Public Member Functions inherited from Component
static const char *	TypeToMarker (ComponentType t)
	Converts component type to marker,. More...
static ComponentType	MarkerToType (const std::string &s)
	Converts marker to component type (case insensitive),. More...
static Component *	Init (const std::string &conf_line)
	Initialize component from a line in config file,. More...
static Component *	Read (std::istream &is, bool binary)
	Read the component from a stream (static method),. More...
Static Public Attributes inherited from Component
static const struct key_value	kMarkerMap []
	The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
Protected Attributes inherited from UpdatableComponent
NnetTrainOptions	opts_
	Option-class with training hyper-parameters,. More...
BaseFloat	learn_rate_coef_
	Scalar applied to learning rate for weight matrices (to be used in ::Update method),. More...
BaseFloat	bias_learn_rate_coef_
	Scalar applied to learning rate for bias (to be used in ::Update method),. More...
Protected Attributes inherited from Component
int32	input_dim_
	Data members,. More...
int32	output_dim_
	Dimension of the output of the Component,. More...

Detailed Description

Definition at line 33 of file nnet-affine-transform.h.

Constructor & Destructor Documentation

AffineTransform()

AffineTransform ( int32  dim_in, int32  dim_out  )

inline

Definition at line 35 of file nnet-affine-transform.h.

Referenced by AffineTransform::Copy().

                                              :
    UpdatableComponent(dim_in, dim_out),
    linearity_(dim_out, dim_in), bias_(dim_out),
    linearity_corr_(dim_out, dim_in), bias_corr_(dim_out),
    max_norm_(0.0)
  { }

~AffineTransform()

~AffineTransform ( )

inline

Definition at line 41 of file nnet-affine-transform.h.

  { }

Member Function Documentation

BackpropagateFnc()

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in, const CuMatrixBase< BaseFloat > &  out, const CuMatrixBase< BaseFloat > &  out_diff, CuMatrixBase< BaseFloat > *  in_diff  )

inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 170 of file nnet-affine-transform.h.

References CuMatrixBase< Real >::AddMatMat(), kaldi::kNoTrans, and AffineTransform::linearity_.

                                                          {
    // multiply error derivative by weights
    in_diff->AddMatMat(1.0, out_diff, kNoTrans, linearity_, kNoTrans, 0.0);
  }

Copy()

Component* Copy ( ) const

inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 44 of file nnet-affine-transform.h.

References AffineTransform::AffineTransform().

{ return new AffineTransform(*this); }

GetBias()

const CuVectorBase<BaseFloat>& GetBias ( ) const

inline

Accessors to the component parameters,.

Definition at line 219 of file nnet-affine-transform.h.

References AffineTransform::bias_.

Referenced by kaldi::ConvertAffineTransformComponent().

{ return bias_; }

GetGradient()

void GetGradient ( VectorBase< BaseFloat > *  gradient ) const

inlinevirtual

Get gradient reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 124 of file nnet-affine-transform.h.

References AffineTransform::bias_, AffineTransform::bias_corr_, VectorBase< Real >::Dim(), KALDI_ASSERT, AffineTransform::linearity_, AffineTransform::linearity_corr_, AffineTransform::NumParams(), and VectorBase< Real >::Range().

                                                          {
    KALDI_ASSERT(gradient->Dim() == NumParams());
    int32 linearity_num_elem = linearity_.NumRows() * linearity_.NumCols();
    gradient->Range(0, linearity_num_elem).CopyRowsFromMat(linearity_corr_);
    gradient->Range(linearity_num_elem, bias_.Dim()).CopyFromVec(bias_corr_);
  }

GetLinearity()

const CuMatrixBase<BaseFloat>& GetLinearity ( ) const

inline

Definition at line 226 of file nnet-affine-transform.h.

References AffineTransform::linearity_.

Referenced by kaldi::ConvertAffineTransformComponent().

{ return linearity_; }

GetParams()

void GetParams ( VectorBase< BaseFloat > *  params ) const

inlinevirtual

Get the trainable parameters reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 131 of file nnet-affine-transform.h.

References AffineTransform::bias_, VectorBase< Real >::Dim(), KALDI_ASSERT, AffineTransform::linearity_, AffineTransform::NumParams(), and VectorBase< Real >::Range().

                                                      {
    KALDI_ASSERT(params->Dim() == NumParams());
    int32 linearity_num_elem = linearity_.NumRows() * linearity_.NumCols();
    params->Range(0, linearity_num_elem).CopyRowsFromMat(linearity_);
    params->Range(linearity_num_elem, bias_.Dim()).CopyFromVec(bias_);
  }

GetType()

ComponentType GetType ( ) const

inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 45 of file nnet-affine-transform.h.

References Component::kAffineTransform.

{ return kAffineTransform; }

Info()

std::string Info ( ) const

inlinevirtual

Print some additional info (after <ComponentName> and the dims),.

Reimplemented from Component.

Definition at line 145 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, kaldi::nnet1::MomentStatistics(), and kaldi::nnet1::ToString().

                         {
    return std::string("\n  linearity") +
      MomentStatistics(linearity_) +
      ", lr-coef " + ToString(learn_rate_coef_) +
      ", max-norm " + ToString(max_norm_) +
      "\n  bias" + MomentStatistics(bias_) +
      ", lr-coef " + ToString(bias_learn_rate_coef_);
  }

InfoGradient()

std::string InfoGradient ( ) const

inlinevirtual

Print some additional info about gradient (after <...> and dims),.

Reimplemented from Component.

Definition at line 153 of file nnet-affine-transform.h.

References AffineTransform::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_corr_, AffineTransform::max_norm_, kaldi::nnet1::MomentStatistics(), and kaldi::nnet1::ToString().

                                 {
    return std::string("\n  linearity_grad") +
      MomentStatistics(linearity_corr_) +
      ", lr-coef " + ToString(learn_rate_coef_) +
      ", max-norm " + ToString(max_norm_) +
      "\n  bias_grad" + MomentStatistics(bias_corr_) +
      ", lr-coef " + ToString(bias_learn_rate_coef_);
  }

InitData()

void InitData ( std::istream &  is )

inlinevirtual

Initialize the content of the component by the 'line' from the prototype,.

Implements UpdatableComponent.

Definition at line 47 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, Component::InputDim(), KALDI_ERR, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, Component::OutputDim(), kaldi::nnet1::RandGauss(), kaldi::nnet1::RandUniform(), kaldi::ReadBasicType(), and kaldi::ReadToken().

                                {
    // define options
    float bias_mean = -2.0, bias_range = 2.0, param_stddev = 0.1;
    // parse config
    std::string token;
    while (is >> std::ws, !is.eof()) {
      ReadToken(is, false, &token);
       if (token == "<ParamStddev>") ReadBasicType(is, false, &param_stddev);
      else if (token == "<BiasMean>")    ReadBasicType(is, false, &bias_mean);
      else if (token == "<BiasRange>")   ReadBasicType(is, false, &bias_range);
      else if (token == "<LearnRateCoef>") ReadBasicType(is, false, &learn_rate_coef_);
      else if (token == "<BiasLearnRateCoef>") ReadBasicType(is, false, &bias_learn_rate_coef_);
      else if (token == "<MaxNorm>") ReadBasicType(is, false, &max_norm_);
      else KALDI_ERR << "Unknown token " << token << ", a typo in config?"
                     << " (ParamStddev|BiasMean|BiasRange|LearnRateCoef|BiasLearnRateCoef)";
    }

    //
    // Initialize trainable parameters,
    //
    // Gaussian with given std_dev (mean = 0),
    linearity_.Resize(OutputDim(), InputDim());
    RandGauss(0.0, param_stddev, &linearity_);
    // Uniform,
    bias_.Resize(OutputDim());
    RandUniform(bias_mean, bias_range, &bias_);
  }

NumParams()

int32 NumParams ( ) const

inlinevirtual

Number of trainable parameters,.

Implements UpdatableComponent.

Definition at line 120 of file nnet-affine-transform.h.

References AffineTransform::bias_, and AffineTransform::linearity_.

Referenced by AffineTransform::GetGradient(), AffineTransform::GetParams(), and AffineTransform::SetParams().

                          {
    return linearity_.NumRows()*linearity_.NumCols() + bias_.Dim();
  }

PropagateFnc()

void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in, CuMatrixBase< BaseFloat > *  out  )

inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 162 of file nnet-affine-transform.h.

References CuMatrixBase< Real >::AddMatMat(), CuMatrixBase< Real >::AddVecToRows(), AffineTransform::bias_, kaldi::kNoTrans, kaldi::kTrans, and AffineTransform::linearity_.

                                                  {
    // precopy bias
    out->AddVecToRows(1.0, bias_, 0.0);
    // multiply by weights^t
    out->AddMatMat(1.0, in, kNoTrans, linearity_, kTrans, 1.0);
  }

ReadData()

void ReadData ( std::istream &  is, bool  binary  )

inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 75 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, kaldi::ExpectToken(), Component::input_dim_, KALDI_ASSERT, KALDI_ERR, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, Component::output_dim_, kaldi::Peek(), kaldi::PeekToken(), kaldi::ReadBasicType(), and kaldi::ReadToken().

                                             {
    // Read all the '<Tokens>' in arbitrary order,
    while ('<' == Peek(is, binary)) {
      int first_char = PeekToken(is, binary);
      switch (first_char) {
        case 'L': ExpectToken(is, binary, "<LearnRateCoef>");
          ReadBasicType(is, binary, &learn_rate_coef_);
          break;
        case 'B': ExpectToken(is, binary, "<BiasLearnRateCoef>");
          ReadBasicType(is, binary, &bias_learn_rate_coef_);
          break;
        case 'M': ExpectToken(is, binary, "<MaxNorm>");
          ReadBasicType(is, binary, &max_norm_);
          break;
        default:
          std::string token;
          ReadToken(is, false, &token);
          KALDI_ERR << "Unknown token: " << token;
      }
    }
    // Read the data (data follow the tokens),

    // weight matrix,
    linearity_.Read(is, binary);
    // bias vector,
    bias_.Read(is, binary);

    KALDI_ASSERT(linearity_.NumRows() == output_dim_);
    KALDI_ASSERT(linearity_.NumCols() == input_dim_);
    KALDI_ASSERT(bias_.Dim() == output_dim_);
  }

SetBias()

void SetBias ( const CuVectorBase< BaseFloat > &  bias )

inline

Definition at line 221 of file nnet-affine-transform.h.

References AffineTransform::bias_, CuVectorBase< Real >::Dim(), and KALDI_ASSERT.

                                                    {
    KALDI_ASSERT(bias.Dim() == bias_.Dim());
    bias_.CopyFromVec(bias);
  }

SetLinearity()

void SetLinearity ( const CuMatrixBase< BaseFloat > &  linearity )

inline

Definition at line 228 of file nnet-affine-transform.h.

References KALDI_ASSERT, AffineTransform::linearity_, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

Referenced by MultiBasisComponent::InitData().

                                                              {
    KALDI_ASSERT(linearity.NumRows() == linearity_.NumRows());
    KALDI_ASSERT(linearity.NumCols() == linearity_.NumCols());
    linearity_.CopyFromMat(linearity);
  }

SetParams()

void SetParams ( const VectorBase< BaseFloat > &  params )

inlinevirtual

Set the trainable parameters from, reshaped as a vector,.

Implements UpdatableComponent.

Definition at line 138 of file nnet-affine-transform.h.

References AffineTransform::bias_, VectorBase< Real >::Dim(), KALDI_ASSERT, AffineTransform::linearity_, AffineTransform::NumParams(), and VectorBase< Real >::Range().

                                                      {
    KALDI_ASSERT(params.Dim() == NumParams());
    int32 linearity_num_elem = linearity_.NumRows() * linearity_.NumCols();
    linearity_.CopyRowsFromVec(params.Range(0, linearity_num_elem));
    bias_.CopyFromVec(params.Range(linearity_num_elem, bias_.Dim()));
  }

Update()

void Update ( const CuMatrixBase< BaseFloat > &  input, const CuMatrixBase< BaseFloat > &  diff  )

inlinevirtual

Compute gradient and update parameters,.

Implements UpdatableComponent.

Definition at line 179 of file nnet-affine-transform.h.

References CuVectorBase< Real >::AddColSumMat(), CuVectorBase< Real >::ApplyFloor(), CuVectorBase< Real >::ApplyPow(), AffineTransform::bias_, AffineTransform::bias_corr_, UpdatableComponent::bias_learn_rate_coef_, CuVectorBase< Real >::InvertElements(), kaldi::kNoTrans, kaldi::kTrans, NnetTrainOptions::l1_penalty, NnetTrainOptions::l2_penalty, NnetTrainOptions::learn_rate, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::linearity_corr_, AffineTransform::max_norm_, NnetTrainOptions::momentum, CuMatrixBase< Real >::MulElements(), CuMatrixBase< Real >::NumRows(), UpdatableComponent::opts_, Component::OutputDim(), kaldi::cu::RegularizeL1(), and CuVectorBase< Real >::Scale().

                                                   {
    // we use following hyperparameters from the option class
    const BaseFloat lr = opts_.learn_rate * learn_rate_coef_;
    const BaseFloat lr_bias = opts_.learn_rate * bias_learn_rate_coef_;
    const BaseFloat mmt = opts_.momentum;
    const BaseFloat l2 = opts_.l2_penalty;
    const BaseFloat l1 = opts_.l1_penalty;
    // we will also need the number of frames in the mini-batch
    const int32 num_frames = input.NumRows();
    // compute gradient (incl. momentum)
    linearity_corr_.AddMatMat(1.0, diff, kTrans, input, kNoTrans, mmt);
    bias_corr_.AddRowSumMat(1.0, diff, mmt);
    // l2 regularization
    if (l2 != 0.0) {
      linearity_.AddMat(-lr*l2*num_frames, linearity_);
    }
    // l1 regularization
    if (l1 != 0.0) {
      cu::RegularizeL1(&linearity_, &linearity_corr_, lr*l1*num_frames, lr);
    }
    // update
    linearity_.AddMat(-lr, linearity_corr_);
    bias_.AddVec(-lr_bias, bias_corr_);
    // max-norm
    if (max_norm_ > 0.0) {
      CuMatrix<BaseFloat> lin_sqr(linearity_);
      lin_sqr.MulElements(linearity_);
      CuVector<BaseFloat> l2(OutputDim());
      l2.AddColSumMat(1.0, lin_sqr, 0.0);
      l2.ApplyPow(0.5);  // we have per-neuron L2 norms,
      CuVector<BaseFloat> scl(l2);
      scl.Scale(1.0/max_norm_);
      scl.ApplyFloor(1.0);
      scl.InvertElements();
      linearity_.MulRowsVec(scl);  // shink to sphere!
    }
  }

WriteData()

void WriteData ( std::ostream &  os, bool  binary  ) const

inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 107 of file nnet-affine-transform.h.

References AffineTransform::bias_, UpdatableComponent::bias_learn_rate_coef_, UpdatableComponent::learn_rate_coef_, AffineTransform::linearity_, AffineTransform::max_norm_, kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                    {
    WriteToken(os, binary, "<LearnRateCoef>");
    WriteBasicType(os, binary, learn_rate_coef_);
    WriteToken(os, binary, "<BiasLearnRateCoef>");
    WriteBasicType(os, binary, bias_learn_rate_coef_);
    WriteToken(os, binary, "<MaxNorm>");
    WriteBasicType(os, binary, max_norm_);
    if (!binary) os << "\n";
    // weights
    linearity_.Write(os, binary);
    bias_.Write(os, binary);
  }

Member Data Documentation

bias_

CuVector<BaseFloat> bias_

private

Definition at line 236 of file nnet-affine-transform.h.

Referenced by AffineTransform::GetBias(), AffineTransform::GetGradient(), AffineTransform::GetParams(), AffineTransform::Info(), AffineTransform::InitData(), AffineTransform::NumParams(), AffineTransform::PropagateFnc(), AffineTransform::ReadData(), AffineTransform::SetBias(), AffineTransform::SetParams(), AffineTransform::Update(), and AffineTransform::WriteData().

bias_corr_

CuVector<BaseFloat> bias_corr_

private

Definition at line 239 of file nnet-affine-transform.h.

Referenced by AffineTransform::GetGradient(), AffineTransform::InfoGradient(), and AffineTransform::Update().

linearity_

CuMatrix<BaseFloat> linearity_

private

Definition at line 235 of file nnet-affine-transform.h.

Referenced by AffineTransform::BackpropagateFnc(), AffineTransform::GetGradient(), AffineTransform::GetLinearity(), AffineTransform::GetParams(), AffineTransform::Info(), AffineTransform::InitData(), AffineTransform::NumParams(), AffineTransform::PropagateFnc(), AffineTransform::ReadData(), AffineTransform::SetLinearity(), AffineTransform::SetParams(), AffineTransform::Update(), and AffineTransform::WriteData().

linearity_corr_

CuMatrix<BaseFloat> linearity_corr_

private

Definition at line 238 of file nnet-affine-transform.h.

Referenced by AffineTransform::GetGradient(), AffineTransform::InfoGradient(), and AffineTransform::Update().

max_norm_

BaseFloat max_norm_

private

Definition at line 241 of file nnet-affine-transform.h.

Referenced by AffineTransform::Info(), AffineTransform::InfoGradient(), AffineTransform::InitData(), AffineTransform::ReadData(), AffineTransform::Update(), and AffineTransform::WriteData().

---

The documentation for this class was generated from the following file:

• nnet/nnet-affine-transform.h