Rbm Class Reference

#include <nnet-rbm.h>

Inheritance diagram for Rbm:

Collaboration diagram for Rbm:

Public Member Functions
	Rbm (int32 dim_in, int32 dim_out)
	~Rbm ()
Component *	Copy () const
	Copy component (deep copy),. More...
ComponentType	GetType () const
	Get Type Identification of the component,. More...
void	InitData (std::istream &is)
	Virtual interface for initialization and I/O,. More...
void	ReadData (std::istream &is, bool binary)
	Reads the component content. More...
void	WriteData (std::ostream &os, bool binary) const
	Writes the component content. More...
void	PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
	Abstract interface for propagation/backpropagation. More...
void	Reconstruct (const CuMatrixBase< BaseFloat > &hid_state, CuMatrix< BaseFloat > *vis_probs)
void	RbmUpdate (const CuMatrixBase< BaseFloat > &pos_vis, const CuMatrixBase< BaseFloat > &pos_hid, const CuMatrixBase< BaseFloat > &neg_vis, const CuMatrixBase< BaseFloat > &neg_hid)
RbmNodeType	VisType () const
RbmNodeType	HidType () const
void	WriteAsNnet (std::ostream &os, bool binary) const
Public Member Functions inherited from RbmBase
	RbmBase (int32 dim_in, int32 dim_out)
void	SetRbmTrainOptions (const RbmTrainOptions &opts)
	Set training hyper-parameters to the network and its UpdatableComponent(s) More...
const RbmTrainOptions &	GetRbmTrainOptions () const
	Get training hyper-parameters from the network. More...
Public Member Functions inherited from Component
	Component (int32 input_dim, int32 output_dim)
	Generic interface of a component,. More...
virtual	~Component ()
virtual bool	IsUpdatable () const
	Check if componeny has 'Updatable' interface (trainable components),. More...
virtual bool	IsMultistream () const
	Check if component has 'Recurrent' interface (trainable and recurrent),. More...
int32	InputDim () const
	Get the dimension of the input,. More...
int32	OutputDim () const
	Get the dimension of the output,. More...
void	Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
	Perform forward-pass propagation 'in' -> 'out',. More...
void	Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
	Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
void	Write (std::ostream &os, bool binary) const
	Write the component to a stream,. More...
virtual std::string	Info () const
	Print some additional info (after <ComponentName> and the dims),. More...
virtual std::string	InfoGradient () const
	Print some additional info about gradient (after <...> and dims),. More...

Protected Attributes
CuMatrix< BaseFloat >	vis_hid_
	Matrix with neuron weights. More...
CuVector< BaseFloat >	vis_bias_
	Vector with biases. More...
CuVector< BaseFloat >	hid_bias_
	Vector with biases. More...
CuMatrix< BaseFloat >	vis_hid_corr_
	Matrix for linearity updates. More...
CuVector< BaseFloat >	vis_bias_corr_
	Vector for bias updates. More...
CuVector< BaseFloat >	hid_bias_corr_
	Vector for bias updates. More...
RbmNodeType	vis_type_
RbmNodeType	hid_type_
Protected Attributes inherited from RbmBase
RbmTrainOptions	rbm_opts_
Protected Attributes inherited from Component
int32	input_dim_
	Data members,. More...
int32	output_dim_
	Dimension of the output of the Component,. More...

Additional Inherited Members
Public Types inherited from RbmBase
enum	RbmNodeType { Bernoulli, Gaussian }
Public Types inherited from Component
enum	ComponentType {   kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,   kConvolutionalComponent, kLstmProjected, kBlstmProjected, kRecurrentComponent,   kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax, kBlockSoftmax,   kSigmoid, kTanh, kParametricRelu, kDropout,   kLengthNormComponent, kTranform = 0x0400, kRbm, kSplice,   kCopy, kTranspose, kBlockLinearity, kAddShift,   kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent, kSimpleSentenceAveragingComponent,   kAveragePoolingComponent, kMaxPoolingComponent, kFramePoolingComponent, kParallelComponent,   kMultiBasisComponent }
	Component type identification mechanism,. More...
Static Public Member Functions inherited from Component
static const char *	TypeToMarker (ComponentType t)
	Converts component type to marker,. More...
static ComponentType	MarkerToType (const std::string &s)
	Converts marker to component type (case insensitive),. More...
static Component *	Init (const std::string &conf_line)
	Initialize component from a line in config file,. More...
static Component *	Read (std::istream &is, bool binary)
	Read the component from a stream (static method),. More...
Static Public Attributes inherited from Component
static const struct key_value	kMarkerMap []
	The table with pairs of Component types and markers (defined in nnet-component.cc),. More...

Detailed Description

Definition at line 96 of file nnet-rbm.h.

Constructor & Destructor Documentation

Rbm()

Rbm ( int32  dim_in, int32  dim_out  )

inline

Definition at line 98 of file nnet-rbm.h.

                                  :
    RbmBase(dim_in, dim_out)
  { }

~Rbm()

~Rbm ( )

inline

Definition at line 102 of file nnet-rbm.h.

  { }

Member Function Documentation

Copy()

Component* Copy ( ) const

inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 105 of file nnet-rbm.h.

                          {
    return new Rbm(*this);
  }

GetType()

ComponentType GetType ( ) const

inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 109 of file nnet-rbm.h.

References Component::kRbm.

                                {
    return kRbm;
  }

HidType()

RbmNodeType HidType ( ) const

inlinevirtual

Implements RbmBase.

Definition at line 393 of file nnet-rbm.h.

                              {
    return hid_type_;
  }

InitData()

void InitData ( std::istream &  is )

inlinevirtual

Virtual interface for initialization and I/O,.

Initialize internal data of a component

Reimplemented from Component.

Definition at line 113 of file nnet-rbm.h.

References RbmBase::Bernoulli, rnnlm::d, RbmBase::Gaussian, Nnet::GetComponent(), Nnet::InputDim(), Component::InputDim(), KALDI_ASSERT, KALDI_ERR, KALDI_LOG, kaldi::Log(), Component::OutputDim(), kaldi::nnet1::RandGauss(), kaldi::nnet1::RandUniform(), Nnet::Read(), kaldi::ReadBasicType(), and kaldi::ReadToken().

                                {
    // define options,
    std::string vis_type;
    std::string hid_type;
    float vis_bias_mean = 0.0, vis_bias_range = 0.0,
          hid_bias_mean = 0.0, hid_bias_range = 0.0,
          param_stddev = 0.1;
    std::string vis_bias_cmvn_file;  // initialize biases to logit(p_active)
    // parse config,
    std::string token;
    while (is >> std::ws, !is.eof()) {
      ReadToken(is, false, &token);
       if (token == "<VisibleType>") ReadToken(is, false, &vis_type);
      else if (token == "<HiddenType>") ReadToken(is, false, &hid_type);
      else if (token == "<VisibleBiasMean>") ReadBasicType(is, false, &vis_bias_mean);
      else if (token == "<VisibleBiasRange>") ReadBasicType(is, false, &vis_bias_range);
      else if (token == "<HiddenBiasMean>") ReadBasicType(is, false, &hid_bias_mean);
      else if (token == "<HiddenBiasRange>") ReadBasicType(is, false, &hid_bias_range);
      else if (token == "<ParamStddev>") ReadBasicType(is, false, &param_stddev);
      else if (token == "<VisibleBiasCmvnFilename>") ReadToken(is, false, &vis_bias_cmvn_file);
      else KALDI_ERR << "Unknown token " << token << " Typo in config?";
    }

    // Translate the 'node' types,
    if (vis_type == "bern" || vis_type == "Bernoulli") vis_type_ = RbmBase::Bernoulli;
    else if (vis_type == "gauss" || vis_type == "Gaussian") vis_type_ = RbmBase::Gaussian;
    else KALDI_ERR << "Wrong <VisibleType>" << vis_type;
    //
    if (hid_type == "bern" || hid_type == "Bernoulli") hid_type_ = RbmBase::Bernoulli;
    else if (hid_type == "gauss" || hid_type == "Gaussian") hid_type_ = RbmBase::Gaussian;
    else KALDI_ERR << "Wrong <HiddenType>" << hid_type;

    //
    // Initialize trainable parameters,
    //
    // visible-hidden connections,
    vis_hid_.Resize(OutputDim(), InputDim());
    RandGauss(0.0, param_stddev, &vis_hid_);
    // hidden-bias,
    hid_bias_.Resize(OutputDim());
    RandUniform(hid_bias_mean, hid_bias_range, &hid_bias_);
    // visible-bias,
    if (vis_bias_cmvn_file == "") {
      vis_bias_.Resize(InputDim());
      RandUniform(vis_bias_mean, vis_bias_range, &vis_bias_);
    } else {
      KALDI_LOG << "Initializing from <VisibleBiasCmvnFilename> "
                << vis_bias_cmvn_file;
      // Reading Nnet with 'global-cmvn' components,
      Nnet cmvn;
      cmvn.Read(vis_bias_cmvn_file);
      KALDI_ASSERT(InputDim() == cmvn.InputDim());
      // The parameters from <AddShift> correspond to 'negative' mean values,
      Vector<BaseFloat> p(cmvn.InputDim());
      dynamic_cast<AddShift&>(cmvn.GetComponent(0)).GetParams(&p);
      p.Scale(-1.0);  // 'un-do' negation of mean values,
      p.ApplyFloor(0.0001);
      p.ApplyCeiling(0.9999);
      // Getting the logit,
      Vector<BaseFloat> logit_p(p.Dim());
      for (int32 d = 0; d < p.Dim(); d++) {
        logit_p(d) = Log(p(d)) - Log(1.0 - p(d));
      }
      vis_bias_ = logit_p;
      KALDI_ASSERT(vis_bias_.Dim() == InputDim());
    }
  }

PropagateFnc()

void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in, CuMatrixBase< BaseFloat > *  out  )

inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 226 of file nnet-rbm.h.

References CuMatrixBase< Real >::AddMatMat(), CuMatrixBase< Real >::AddVecToRows(), RbmBase::Bernoulli, kaldi::kNoTrans, kaldi::kTrans, and CuMatrixBase< Real >::Sigmoid().

                                                  {
    // pre-fill with bias
    out->AddVecToRows(1.0, hid_bias_, 0.0);
    // multiply by weights^t
    out->AddMatMat(1.0, in, kNoTrans, vis_hid_, kTrans, 1.0);
    // optionally apply sigmoid
    if (hid_type_ == RbmBase::Bernoulli) {
      out->Sigmoid(*out);
    }
  }

RbmUpdate()

void RbmUpdate ( const CuMatrixBase< BaseFloat > &  pos_vis, const CuMatrixBase< BaseFloat > &  pos_hid, const CuMatrixBase< BaseFloat > &  neg_vis, const CuMatrixBase< BaseFloat > &  neg_hid  )

inlinevirtual

Implements RbmBase.

Definition at line 262 of file nnet-rbm.h.

References kaldi::nnet1::CheckNanInf(), kaldi::nnet1::ComputeStdDev(), RbmBase::Gaussian, Component::input_dim_, KALDI_ASSERT, KALDI_WARN, kaldi::kNoTrans, kaldi::kSetZero, kaldi::kTrans, RbmTrainOptions::l2_penalty, RbmTrainOptions::learn_rate, RbmTrainOptions::momentum, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), Component::output_dim_, and RbmBase::rbm_opts_.

                                                         {
    // dims
    KALDI_ASSERT(pos_vis.NumRows() == pos_hid.NumRows() &&
           pos_vis.NumRows() == neg_vis.NumRows() &&
           pos_vis.NumRows() == neg_hid.NumRows() &&
           pos_vis.NumCols() == neg_vis.NumCols() &&
           pos_hid.NumCols() == neg_hid.NumCols() &&
           pos_vis.NumCols() == input_dim_ &&
           pos_hid.NumCols() == output_dim_);

    // lazy initialization of buffers
    if ( vis_hid_corr_.NumRows() != vis_hid_.NumRows() ||
         vis_hid_corr_.NumCols() != vis_hid_.NumCols() ||
         vis_bias_corr_.Dim()    != vis_bias_.Dim()    ||
         hid_bias_corr_.Dim()    != hid_bias_.Dim()     ) {
      vis_hid_corr_.Resize(vis_hid_.NumRows(), vis_hid_.NumCols(), kSetZero);
      vis_bias_corr_.Resize(vis_bias_.Dim(), kSetZero);
      hid_bias_corr_.Resize(hid_bias_.Dim(), kSetZero);
    }

    // ANTI-WEIGHT-EXPLOSION PROTECTION (Gaussian-Bernoulli RBM)
    //
    // in the following section we detect that the weights in
    // Gaussian-Bernoulli RBM are almost exploding. The weight
    // explosion is caused by large variance of the reconstructed data,
    // which causes a feed-back loop that keeps increasing the weights.
    //
    // To avoid explosion, the standard-deviation of the visible-data
    // and reconstructed-data should be about the same.
    // The model is particularly sensitive at the very
    // beginning of the CD-1 training.
    //
    // We compute the standard deviations on
    // * 'A' : input mini-batch
    // * 'B' : reconstruction.
    // When 'B > 2*A', we stabilize the training in this way:
    // 1. we scale down the weights and biases by 'A/B',
    // 2. we shrink learning rate by 0.9x,
    // 3. we reset the momentum buffer,
    //
    // A warning message is put to the log. In later stage
    // the learning-rate returns back to its original value.
    //
    // To avoid the issue, we make sure that the weight-matrix
    // is sensibly initialized.
    //
    if (vis_type_ == RbmBase::Gaussian) {
      // check the data have no nan/inf:
      CheckNanInf(pos_vis, "pos_vis");
      CheckNanInf(pos_hid, "pos_hid");
      CheckNanInf(neg_vis, "neg_vis");
      CheckNanInf(neg_hid, "pos_hid");

      // get standard deviations of pos_vis and neg_vis:
      BaseFloat pos_vis_std = ComputeStdDev(pos_vis);
      BaseFloat neg_vis_std = ComputeStdDev(neg_vis);

      // monitor the standard deviation mismatch : data vs. reconstruction
      if (pos_vis_std * 2 < neg_vis_std) {
        // 1) scale-down the weights and biases
        BaseFloat scale = pos_vis_std / neg_vis_std;
        vis_hid_.Scale(scale);
        vis_bias_.Scale(scale);
        hid_bias_.Scale(scale);
        // 2) reduce the learning rate
        rbm_opts_.learn_rate *= 0.9;
        // 3) reset the momentum buffers
        vis_hid_corr_.SetZero();
        vis_bias_corr_.SetZero();
        hid_bias_corr_.SetZero();

        KALDI_WARN << "Mismatch between pos_vis and neg_vis variances, "
                   << "danger of weight explosion."
                   << " a) Reducing weights with scale " << scale
                   << " b) Lowering learning rate to " << rbm_opts_.learn_rate
                   << " [pos_vis_std:" << pos_vis_std
                   << ",neg_vis_std:" << neg_vis_std << "]";
        return; /* i.e. don't update now, the update would be too BIG */
      }
    }
    //
    // End of weight-explosion check


    //  We use these training hyper-parameters
    //
    const BaseFloat lr = rbm_opts_.learn_rate;
    const BaseFloat mmt = rbm_opts_.momentum;
    const BaseFloat l2 = rbm_opts_.l2_penalty;

    //  UPDATE vishid matrix
    //
    //  vishidinc = momentum*vishidinc + ...
    //              epsilonw*( (posprods-negprods)/numcases - weightcost*vishid)
    //
    //  vishidinc[t] = -(epsilonw/numcases)*negprods + momentum*vishidinc[t-1]
    //                 +(epsilonw/numcases)*posprods
    //                 -(epsilonw*weightcost)*vishid[t-1]
    //
    BaseFloat N = static_cast<BaseFloat>(pos_vis.NumRows());
    vis_hid_corr_.AddMatMat(-lr/N, neg_hid, kTrans, neg_vis, kNoTrans, mmt);
    vis_hid_corr_.AddMatMat(+lr/N, pos_hid, kTrans, pos_vis, kNoTrans, 1.0);
    vis_hid_corr_.AddMat(-lr*l2, vis_hid_);
    vis_hid_.AddMat(1.0, vis_hid_corr_);

    //  UPDATE visbias vector
    //
    //  visbiasinc = momentum*visbiasinc +
    //               (epsilonvb/numcases)*(posvisact-negvisact);
    //
    vis_bias_corr_.AddRowSumMat(-lr/N, neg_vis, mmt);
    vis_bias_corr_.AddRowSumMat(+lr/N, pos_vis, 1.0);
    vis_bias_.AddVec(1.0, vis_bias_corr_, 1.0);

    //  UPDATE hidbias vector
    //
    // hidbiasinc = momentum*hidbiasinc +
    //              (epsilonhb/numcases)*(poshidact-neghidact);
    //
    hid_bias_corr_.AddRowSumMat(-lr/N, neg_hid, mmt);
    hid_bias_corr_.AddRowSumMat(+lr/N, pos_hid, 1.0);
    hid_bias_.AddVec(1.0, hid_bias_corr_, 1.0);
  }

ReadData()

void ReadData ( std::istream &  is, bool  binary  )

inlinevirtual

Reads the component content.

Reimplemented from Component.

Definition at line 182 of file nnet-rbm.h.

References RbmBase::Bernoulli, RbmBase::Gaussian, Component::input_dim_, KALDI_ASSERT, Component::output_dim_, and kaldi::ReadToken().

                                             {
    std::string vis_node_type, hid_node_type;
    ReadToken(is, binary, &vis_node_type);
    ReadToken(is, binary, &hid_node_type);

    if (vis_node_type == "bern") {
      vis_type_ = RbmBase::Bernoulli;
    } else if (vis_node_type == "gauss") {
      vis_type_ = RbmBase::Gaussian;
    }
    if (hid_node_type == "bern") {
      hid_type_ = RbmBase::Bernoulli;
    } else if (hid_node_type == "gauss") {
      hid_type_ = RbmBase::Gaussian;
    }

    vis_hid_.Read(is, binary);
    vis_bias_.Read(is, binary);
    hid_bias_.Read(is, binary);

    KALDI_ASSERT(vis_hid_.NumRows() == output_dim_);
    KALDI_ASSERT(vis_hid_.NumCols() == input_dim_);
    KALDI_ASSERT(vis_bias_.Dim() == input_dim_);
    KALDI_ASSERT(hid_bias_.Dim() == output_dim_);
  }

Reconstruct()

void Reconstruct ( const CuMatrixBase< BaseFloat > &  hid_state, CuMatrix< BaseFloat > *  vis_probs  )

inlinevirtual

Implements RbmBase.

Definition at line 239 of file nnet-rbm.h.

References CuMatrixBase< Real >::AddMatMat(), CuMatrixBase< Real >::AddVecToRows(), RbmBase::Bernoulli, Component::input_dim_, KALDI_ERR, kaldi::kNoTrans, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), Component::output_dim_, CuMatrix< Real >::Resize(), and CuMatrixBase< Real >::Sigmoid().

                                                   {
    // check the dim
    if (output_dim_ != hid_state.NumCols()) {
      KALDI_ERR << "Nonmatching dims, component:" << output_dim_
                << " data:" << hid_state.NumCols();
    }
    // optionally allocate buffer
    if (input_dim_ != vis_probs->NumCols() ||
        hid_state.NumRows() != vis_probs->NumRows()) {
      vis_probs->Resize(hid_state.NumRows(), input_dim_);
    }

    // pre-fill with bias
    vis_probs->AddVecToRows(1.0, vis_bias_, 0.0);
    // multiply by weights
    vis_probs->AddMatMat(1.0, hid_state, kNoTrans, vis_hid_, kNoTrans, 1.0);
    // optionally apply sigmoid
    if (vis_type_ == RbmBase::Bernoulli) {
      vis_probs->Sigmoid(*vis_probs);
    }
  }

VisType()

RbmNodeType VisType ( ) const

inlinevirtual

Implements RbmBase.

Definition at line 389 of file nnet-rbm.h.

                              {
    return vis_type_;
  }

WriteAsNnet()

void WriteAsNnet ( std::ostream &  os, bool  binary  ) const

inlinevirtual

Implements RbmBase.

Definition at line 397 of file nnet-rbm.h.

References RbmBase::Bernoulli, RbmBase::HidType(), Component::InputDim(), Component::kAffineTransform, Component::kSigmoid, Component::OutputDim(), Component::TypeToMarker(), kaldi::WriteBasicType(), and kaldi::WriteToken().

                                                      {
    // header,
    WriteToken(os, binary, Component::TypeToMarker(Component::kAffineTransform));
    WriteBasicType(os, binary, OutputDim());
    WriteBasicType(os, binary, InputDim());
    if (!binary) os << "\n";
    // data,
    vis_hid_.Write(os, binary);
    hid_bias_.Write(os, binary);
    // sigmoid activation,
    if (HidType() == Bernoulli) {
      WriteToken(os, binary, Component::TypeToMarker(Component::kSigmoid));
      WriteBasicType(os, binary, OutputDim());
      WriteBasicType(os, binary, OutputDim());
    }
    if (!binary) os << "\n";
  }

WriteData()

void WriteData ( std::ostream &  os, bool  binary  ) const

inlinevirtual

Writes the component content.

Reimplemented from Component.

Definition at line 208 of file nnet-rbm.h.

References RbmBase::Bernoulli, RbmBase::Gaussian, KALDI_ERR, and kaldi::WriteToken().

                                                    {
    switch (vis_type_) {
      case Bernoulli : WriteToken(os,binary, "bern"); break;
      case Gaussian  : WriteToken(os,binary, "gauss"); break;
      default : KALDI_ERR << "Unknown type " << vis_type_;
    }
    switch (hid_type_) {
      case Bernoulli : WriteToken(os,binary, "bern"); break;
      case Gaussian  : WriteToken(os,binary, "gauss"); break;
      default : KALDI_ERR << "Unknown type " << hid_type_;
    }
    vis_hid_.Write(os, binary);
    vis_bias_.Write(os, binary);
    hid_bias_.Write(os, binary);
  }

Member Data Documentation

hid_bias_

CuVector<BaseFloat> hid_bias_

protected

Vector with biases.

Definition at line 418 of file nnet-rbm.h.

hid_bias_corr_

CuVector<BaseFloat> hid_bias_corr_

protected

Vector for bias updates.

Definition at line 422 of file nnet-rbm.h.

hid_type_

RbmNodeType hid_type_

protected

Definition at line 425 of file nnet-rbm.h.

vis_bias_

CuVector<BaseFloat> vis_bias_

protected

Vector with biases.

Definition at line 417 of file nnet-rbm.h.

vis_bias_corr_

CuVector<BaseFloat> vis_bias_corr_

protected

Vector for bias updates.

Definition at line 421 of file nnet-rbm.h.

vis_hid_

CuMatrix<BaseFloat> vis_hid_

protected

Matrix with neuron weights.

Definition at line 416 of file nnet-rbm.h.

vis_hid_corr_

CuMatrix<BaseFloat> vis_hid_corr_

protected

Matrix for linearity updates.

Definition at line 420 of file nnet-rbm.h.

vis_type_

RbmNodeType vis_type_

protected

Definition at line 424 of file nnet-rbm.h.

---

The documentation for this class was generated from the following file:

• nnet/nnet-rbm.h