All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
HiddenSoftmax Class Reference

#include <nnet-activation.h>

Inheritance diagram for HiddenSoftmax:
Collaboration diagram for HiddenSoftmax:

Public Member Functions

 HiddenSoftmax (int32 dim_in, int32 dim_out)
 
 ~HiddenSoftmax ()
 
ComponentCopy () const
 Copy component (deep copy),. More...
 
ComponentType GetType () const
 Get Type Identification of the component,. More...
 
void PropagateFnc (const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out)
 Abstract interface for propagation/backpropagation. More...
 
void BackpropagateFnc (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrixBase< BaseFloat > *in_diff)
 Backward pass transformation (to be implemented by descending class...) More...
 
- Public Member Functions inherited from Component
 Component (int32 input_dim, int32 output_dim)
 Generic interface of a component,. More...
 
virtual ~Component ()
 
virtual bool IsUpdatable () const
 Check if componeny has 'Updatable' interface (trainable components),. More...
 
virtual bool IsMultistream () const
 Check if component has 'Recurrent' interface (trainable and recurrent),. More...
 
int32 InputDim () const
 Get the dimension of the input,. More...
 
int32 OutputDim () const
 Get the dimension of the output,. More...
 
void Propagate (const CuMatrixBase< BaseFloat > &in, CuMatrix< BaseFloat > *out)
 Perform forward-pass propagation 'in' -> 'out',. More...
 
void Backpropagate (const CuMatrixBase< BaseFloat > &in, const CuMatrixBase< BaseFloat > &out, const CuMatrixBase< BaseFloat > &out_diff, CuMatrix< BaseFloat > *in_diff)
 Perform backward-pass propagation 'out_diff' -> 'in_diff'. More...
 
void Write (std::ostream &os, bool binary) const
 Write the component to a stream,. More...
 
virtual std::string Info () const
 Print some additional info (after <ComponentName> and the dims),. More...
 
virtual std::string InfoGradient () const
 Print some additional info about gradient (after <...> and dims),. More...
 

Private Attributes

CuVector< BaseFloatdiag_out_diff_out_
 buffer for dot-products in BackpropagateFnc, More...
 

Additional Inherited Members

- Public Types inherited from Component
enum  ComponentType {
  kUnknown = 0x0, kUpdatableComponent = 0x0100, kAffineTransform, kLinearTransform,
  kConvolutionalComponent, kConvolutional2DComponent, kLstmProjected, kBlstmProjected,
  kRecurrentComponent, kActivationFunction = 0x0200, kSoftmax, kHiddenSoftmax,
  kBlockSoftmax, kSigmoid, kTanh, kParametricRelu,
  kDropout, kLengthNormComponent, kTranform = 0x0400, kRbm,
  kSplice, kCopy, kTranspose, kBlockLinearity,
  kAddShift, kRescale, kKlHmm = 0x0800, kSentenceAveragingComponent,
  kSimpleSentenceAveragingComponent, kAveragePoolingComponent, kAveragePooling2DComponent, kMaxPoolingComponent,
  kMaxPooling2DComponent, kFramePoolingComponent, kParallelComponent, kMultiBasisComponent
}
 Component type identification mechanism,. More...
 
- Static Public Member Functions inherited from Component
static const char * TypeToMarker (ComponentType t)
 Converts component type to marker,. More...
 
static ComponentType MarkerToType (const std::string &s)
 Converts marker to component type (case insensitive),. More...
 
static ComponentInit (const std::string &conf_line)
 Initialize component from a line in config file,. More...
 
static ComponentRead (std::istream &is, bool binary)
 Read the component from a stream (static method),. More...
 
- Static Public Attributes inherited from Component
static const struct key_value kMarkerMap []
 The table with pairs of Component types and markers (defined in nnet-component.cc),. More...
 
- Protected Member Functions inherited from Component
virtual void InitData (std::istream &is)
 Virtual interface for initialization and I/O,. More...
 
virtual void ReadData (std::istream &is, bool binary)
 Reads the component content. More...
 
virtual void WriteData (std::ostream &os, bool binary) const
 Writes the component content. More...
 
- Protected Attributes inherited from Component
int32 input_dim_
 Data members,. More...
 
int32 output_dim_
 Dimension of the output of the Component,. More...
 

Detailed Description

Definition at line 69 of file nnet-activation.h.

Constructor & Destructor Documentation

HiddenSoftmax ( int32  dim_in,
int32  dim_out 
)
inline

Definition at line 71 of file nnet-activation.h.

Referenced by HiddenSoftmax::Copy().

71  :
72  Component(dim_in, dim_out)
73  { }
Component(int32 input_dim, int32 output_dim)
Generic interface of a component,.
~HiddenSoftmax ( )
inline

Definition at line 75 of file nnet-activation.h.

76  { }

Member Function Documentation

void BackpropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
const CuMatrixBase< BaseFloat > &  out,
const CuMatrixBase< BaseFloat > &  out_diff,
CuMatrixBase< BaseFloat > *  in_diff 
)
inlinevirtual

Backward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 87 of file nnet-activation.h.

References CuVectorBase< Real >::AddDiagMatMat(), CuMatrixBase< Real >::AddDiagVecMat(), CuMatrixBase< Real >::CopyFromMat(), HiddenSoftmax::diag_out_diff_out_, kaldi::kNoTrans, kaldi::kTrans, CuMatrixBase< Real >::MulElements(), CuMatrixBase< Real >::NumRows(), and CuVector< Real >::Resize().

90  {
91  // This Softmax should be used for a hidden layer, it calculates
92  // the true Jacobian of Softmax: J = diag(out) - out*out^T
93 
94  // The backpropagation formual is:
95  // in_diff = out_diff \odot out - out(out_diff^T * out)
96  // (where \odot is Hadamard product)
97 
98  // 1st term, out_diff \odot out,
99  in_diff->CopyFromMat(out_diff);
100  in_diff->MulElements(out);
101 
102  // 2nd term, -out(out_diff^T * out),
104  diag_out_diff_out_.AddDiagMatMat(1.0, out_diff, kNoTrans, out, kTrans, 0.0);
105  in_diff->AddDiagVecMat(-1.0, diag_out_diff_out_, out, kNoTrans, 1.0);
106  }
void AddDiagVecMat(const Real alpha, const CuVectorBase< Real > &v, const CuMatrixBase< Real > &M, MatrixTransposeType transM, Real beta=1.0)
*this = beta * *this + alpha * diag(v) * M [or M^T].
Definition: cu-matrix.cc:1231
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
void Resize(MatrixIndexT dim, MatrixResizeType t=kSetZero)
Allocate the memory.
Definition: cu-vector.cc:892
void AddDiagMatMat(Real alpha, const CuMatrixBase< Real > &M, MatrixTransposeType transM, const CuMatrixBase< Real > &N, MatrixTransposeType transN, Real beta=1.0)
Add the diagonal of a matrix product: *this = diag(M N), assuming the "trans" arguments are both kNoT...
Definition: cu-vector.cc:544
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
void MulElements(const CuMatrixBase< Real > &A)
Multiply two matrices elementwise: C = C .* A.
Definition: cu-matrix.cc:652
CuVector< BaseFloat > diag_out_diff_out_
buffer for dot-products in BackpropagateFnc,
Component* Copy ( ) const
inlinevirtual

Copy component (deep copy),.

Implements Component.

Definition at line 78 of file nnet-activation.h.

References HiddenSoftmax::HiddenSoftmax().

78 { return new HiddenSoftmax(*this); }
HiddenSoftmax(int32 dim_in, int32 dim_out)
ComponentType GetType ( ) const
inlinevirtual

Get Type Identification of the component,.

Implements Component.

Definition at line 79 of file nnet-activation.h.

References Component::kHiddenSoftmax.

void PropagateFnc ( const CuMatrixBase< BaseFloat > &  in,
CuMatrixBase< BaseFloat > *  out 
)
inlinevirtual

Abstract interface for propagation/backpropagation.

Forward pass transformation (to be implemented by descending class...)

Implements Component.

Definition at line 81 of file nnet-activation.h.

References CuMatrixBase< Real >::ApplySoftMaxPerRow().

82  {
83  // y = e^x_j/sum_j(e^x_j)
84  out->ApplySoftMaxPerRow(in);
85  }
void ApplySoftMaxPerRow(const CuMatrixBase< Real > &src)
Softmax nonlinearity Y = Softmax(X) : Yij = e^Xij / sum_k(e^Xik), done to each row, with attention to avoiding overflow or underflow.
Definition: cu-matrix.cc:1563

Member Data Documentation

CuVector<BaseFloat> diag_out_diff_out_
private

buffer for dot-products in BackpropagateFnc,

Definition at line 110 of file nnet-activation.h.

Referenced by HiddenSoftmax::BackpropagateFnc().


The documentation for this class was generated from the following file: