All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
kaldi::nnet2 Namespace Reference

Classes

class  AdditiveNoiseComponent
 This is a bit similar to dropout but adding (not multiplying) Gaussian noise with a given standard deviation. More...
 
class  AffineComponent
 
class  AffineComponentPreconditioned
 
class  AffineComponentPreconditionedOnline
 Keywords: natural gradient descent, NG-SGD, naturalgradient. More...
 
class  AmNnet
 
class  BlockAffineComponent
 
class  BlockAffineComponentPreconditioned
 
class  ChunkInfo
 ChunkInfo is a class whose purpose is to describe the structure of matrices holding features. More...
 
class  Component
 Abstract class, basic element of the network, it is a box with defined inputs, outputs, and tranformation functions interface. More...
 
class  Convolutional1dComponent
 Convolutional1dComponent implements convolution over frequency axis. More...
 
class  DctComponent
 Discrete cosine transform. More...
 
class  DecodableAmNnet
 DecodableAmNnet is a decodable object that decodes with a neural net acoustic model of type AmNnet. More...
 
class  DecodableAmNnetParallel
 This version of DecodableAmNnet is intended for a version of the decoder that processes different utterances with multiple threads. More...
 
class  DecodableNnet2Online
 This Decodable object for class nnet2::AmNnet takes feature input from class OnlineFeatureInterface, unlike, say, class DecodableAmNnet which takes feature input from a matrix. More...
 
struct  DecodableNnet2OnlineOptions
 
class  DiscriminativeExampleSplitter
 For each frame, judge: More...
 
class  DiscriminativeExamplesRepository
 This struct stores neural net training examples to be used in multi-threaded training. More...
 
struct  DiscriminativeNnetExample
 This struct is used to store the information we need for discriminative training (MMI or MPE). More...
 
class  DiscTrainParallelClass
 
class  DoBackpropParallelClass
 
class  DropoutComponent
 This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two. More...
 
class  ExamplesRepository
 This class stores neural net training examples to be used in multi-threaded training. More...
 
class  FastNnetCombiner
 
class  FisherComputationClass
 
class  FixedAffineComponent
 FixedAffineComponent is an affine transform that is supplied at network initialization time and is not trainable. More...
 
class  FixedBiasComponent
 FixedBiasComponent applies a fixed per-element bias; it's similar to the AddShift component in the nnet1 setup (and only needed for nnet1 model conversion. More...
 
class  FixedLinearComponent
 FixedLinearComponent is a linear transform that is supplied at network initialization time and is not trainable. More...
 
class  FixedScaleComponent
 FixedScaleComponent applies a fixed per-element scale; it's similar to the Rescale component in the nnet1 setup (and only needed for nnet1 model conversion). More...
 
class  LimitRankClass
 
class  LogSoftmaxComponent
 
class  MaxoutComponent
 
class  MaxpoolingComponent
 MaxPoolingComponent : Maxpooling component was firstly used in ConvNet for selecting an representative activation in an area. More...
 
class  Nnet
 
struct  NnetCombineAconfig
 
struct  NnetCombineConfig
 Configuration class that controls neural net combination, where we combine a number of neural nets, trying to find for each layer the optimal weighted combination of the different neural-net parameters. More...
 
struct  NnetCombineFastConfig
 Configuration class that controls neural net combination, where we combine a number of neural nets, trying to find for each layer the optimal weighted combination of the different neural-net parameters. More...
 
class  NnetComputer
 
struct  NnetDiscriminativeStats
 
struct  NnetDiscriminativeUpdateOptions
 
class  NnetDiscriminativeUpdater
 
class  NnetEnsembleTrainer
 
struct  NnetEnsembleTrainerConfig
 
struct  NnetExample
 NnetExample is the input data and corresponding label (or labels) for one or more frames of input, used for standard cross-entropy training of neural nets (and possibly for other objective functions). More...
 
class  NnetExampleBackgroundReader
 
struct  NnetFixConfig
 
struct  NnetLimitRankOpts
 
struct  NnetLogprobTask
 
struct  NnetMixupConfig
 
class  NnetOnlineComputer
 
struct  NnetRescaleConfig
 
class  NnetRescaler
 
struct  NnetShrinkConfig
 Configuration class that controls neural net "shrinkage" which is actually a scaling on the parameters of each of the updatable layers. More...
 
struct  NnetSimpleTrainerConfig
 
class  NnetStats
 
struct  NnetStatsConfig
 
class  NnetUpdater
 
struct  NnetWidenConfig
 Configuration class that controls neural net "widening", which means increasing the dimension of the hidden layers of an already-trained neural net. More...
 
class  NonlinearComponent
 This kind of Component is a base-class for things like sigmoid and softmax. More...
 
class  NormalizeComponent
 
class  OnlinePreconditioner
 Keywords for search: natural gradient, naturalgradient, NG-SGD. More...
 
class  OnlinePreconditionerSimple
 
class  PermuteComponent
 PermuteComponent does a permutation of the dimensions (by default, a fixed random permutation, but it may be specified). More...
 
class  PnormComponent
 
class  PowerComponent
 Take the absoute values of an input vector to a power. More...
 
class  RandomComponent
 
class  RectifiedLinearComponent
 
class  ScaleComponent
 
class  SigmoidComponent
 
class  SoftHingeComponent
 
class  SoftmaxComponent
 
class  SpliceComponent
 Splices a context window of frames together [over time]. More...
 
class  SpliceMaxComponent
 This is as SpliceComponent but outputs the max of any of the inputs (taking the max across time). More...
 
struct  SplitDiscriminativeExampleConfig
 Config structure for SplitExample, for splitting discriminative training examples. More...
 
struct  SplitExampleStats
 This struct exists only for diagnostic purposes. More...
 
class  SumGroupComponent
 
class  TanhComponent
 
class  UpdatableComponent
 Class UpdatableComponent is a Component which has trainable parameters and contains some global parameters for stochastic gradient descent (learning rate, L2 regularization constant). More...
 

Typedefs

typedef TableWriter
< KaldiObjectHolder
< NnetExample > > 
NnetExampleWriter
 
typedef SequentialTableReader
< KaldiObjectHolder
< NnetExample > > 
SequentialNnetExampleReader
 
typedef
RandomAccessTableReader
< KaldiObjectHolder
< NnetExample > > 
RandomAccessNnetExampleReader
 
typedef TableWriter
< KaldiObjectHolder
< DiscriminativeNnetExample > > 
DiscriminativeNnetExampleWriter
 
typedef SequentialTableReader
< KaldiObjectHolder
< DiscriminativeNnetExample > > 
SequentialDiscriminativeNnetExampleReader
 
typedef
RandomAccessTableReader
< KaldiObjectHolder
< DiscriminativeNnetExample > > 
RandomAccessDiscriminativeNnetExampleReader
 

Functions

void UnitTestAmNnet ()
 
static void GetUpdateDirection (const std::vector< Nnet > &nnets, Nnet *direction)
 
static void AddDirection (const Nnet &orig_nnet, const Nnet &direction, const VectorBase< BaseFloat > &scales, Nnet *dest)
 Sets "dest" to orig_nnet plus "direction", with each updatable component of "direction" first scaled by the appropriate scale. More...
 
static BaseFloat ComputeObjfAndGradient (const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const Nnet &orig_nnet, const Nnet &direction, Vector< double > *gradient)
 
void CombineNnetsA (const NnetCombineAconfig &config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)
 
void CombineNnetsFast (const NnetCombineFastConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets_in, Nnet *nnet_out)
 
static void CombineNnets (const Vector< BaseFloat > &scale_params, const std::vector< Nnet > &nnets, Nnet *dest)
 
static int32 GetInitialModel (const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets)
 Returns an integer saying which model to use: either 0 ... More...
 
static void GetInitialScaleParams (const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Vector< double > *scale_params)
 
static double ComputeObjfAndGradient (const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const std::vector< Nnet > &nnets, bool debug, Vector< double > *gradient)
 
void CombineNnets (const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)
 
static void GiveNnetCorrectTopology (Nnet *nnet, AffineComponent **affine_component, SoftmaxComponent **softmax_component, SumGroupComponent **sum_group_component)
 This function makes sure the neural net ends with a SumGroupComponent. More...
 
void MixupNnet (const NnetMixupConfig &mixup_config, Nnet *nnet)
 This function works as follows. More...
 
void UnitTestGenericComponentInternal (const Component &component, const ChunkInfo in_info, const ChunkInfo out_info)
 
void UnitTestGenericComponentInternal (const Component &component)
 
void UnitTestSigmoidComponent ()
 
template<class T >
void UnitTestGenericComponent (std::string extra_str="")
 
void UnitTestMaxoutComponent ()
 
void UnitTestPnormComponent ()
 
void UnitTestMaxpoolingComponent ()
 
void UnitTestAffineComponent ()
 
void UnitTestConvolutional1dComponent ()
 
void UnitTestDropoutComponent ()
 
void UnitTestAdditiveNoiseComponent ()
 
void UnitTestScaleComponent ()
 
void UnitTestAffineComponentPreconditioned ()
 
void UnitTestAffineComponentPreconditionedOnline ()
 
void UnitTestBlockAffineComponent ()
 
void UnitTestBlockAffineComponentPreconditioned ()
 
void UnitTestSumGroupComponent ()
 
void UnitTestDctComponent ()
 
void UnitTestFixedLinearComponent ()
 
void UnitTestFixedAffineComponent ()
 
void UnitTestFixedScaleComponent ()
 
void UnitTestFixedBiasComponent ()
 
void UnitTestParsing ()
 
void UnitTestSpliceComponent ()
 
void BasicDebugTestForSpliceMax (bool output=false)
 
static void ExpectOneOrTwoTokens (std::istream &is, bool binary, const std::string &token1, const std::string &token2)
 
bool ParseFromString (const std::string &name, std::string *string, int32 *param)
 Functions used in Init routines. More...
 
bool ParseFromString (const std::string &name, std::string *string, bool *param)
 This version is for parameters of type bool, which can appear as any string beginning with f, F, t or T. More...
 
bool ParseFromString (const std::string &name, std::string *string, BaseFloat *param)
 This version is for parameters of type BaseFloat. More...
 
bool ParseFromString (const std::string &name, std::string *string, std::string *param)
 
bool ParseFromString (const std::string &name, std::string *string, std::vector< int32 > *param)
 This version is for parameters of type std::vector<int32>; it expects them as a colon-separated list, without spaces. More...
 
void NnetDiscriminativeUpdateParallel (const AmNnet &am_nnet, const TransitionModel &tmodel, const NnetDiscriminativeUpdateOptions &opts, int32 num_threads, SequentialDiscriminativeNnetExampleReader *example_reader, Nnet *nnet_to_update, NnetDiscriminativeStats *stats)
 
void NnetDiscriminativeUpdate (const AmNnet &am_nnet, const TransitionModel &tmodel, const NnetDiscriminativeUpdateOptions &opts, const DiscriminativeNnetExample &eg, Nnet *nnet_to_update, NnetDiscriminativeStats *stats)
 Does the neural net computation, lattice forward-backward, and backprop, for either the MMI, MPFE or SMBR objective functions. More...
 
void UnitTestNnetCompute ()
 
void UnitTestNnetComputeChunked ()
 
void NnetComputation (const Nnet &nnet, const CuMatrixBase< BaseFloat > &input,bool pad_input, CuMatrixBase< BaseFloat > *output)
 Does the basic neural net computation, on a sequence of data (e.g. More...
 
void NnetComputationChunked (const Nnet &nnet, const Matrix< BaseFloat > &input,int32 chunk_size, Matrix< BaseFloat > *output)
 Does the basic neural net computation, on a sequence of data (e.g. More...
 
BaseFloat NnetGradientComputation (const Nnet &nnet, const CuMatrixBase< BaseFloat > &input, bool pad_input, const Posterior &pdf_post, Nnet *nnet_to_update)
 
BaseFloat NnetGradientComputation (const Nnet &nnet, const MatrixBase< BaseFloat > &input, bool pad_input, BaseFloat utterance_weight, const std::vector< int32 > &labels, Nnet *nnet_to_update)
 Does the neural net computation and backprop, given input and labels. More...
 
void UnitTestSolvePackingProblem ()
 
bool LatticeToDiscriminativeExample (const std::vector< int32 > &alignment, const Matrix< BaseFloat > &feats, const CompactLattice &clat, BaseFloat weight, int32 left_context, int32 right_context, DiscriminativeNnetExample *eg)
 Converts lattice to discriminative training example. More...
 
void SplitDiscriminativeExample (const SplitDiscriminativeExampleConfig &config, const TransitionModel &tmodel, const DiscriminativeNnetExample &eg, std::vector< DiscriminativeNnetExample > *egs_out, SplitExampleStats *stats_out)
 Split a "discriminative example" into multiple pieces, splitting where the lattice has "pinch points". More...
 
void ExciseDiscriminativeExample (const SplitDiscriminativeExampleConfig &config, const TransitionModel &tmodel, const DiscriminativeNnetExample &eg, std::vector< DiscriminativeNnetExample > *egs_out, SplitExampleStats *stats_out)
 Remove unnecessary frames from discriminative training example. More...
 
void UpdateHash (const TransitionModel &tmodel, const DiscriminativeNnetExample &eg, std::string criterion, bool drop_frames, bool one_silence_class, Matrix< double > *hash, double *num_weight, double *den_weight, double *tot_t)
 This function is used in code that tests the functionality that we provide here, about splitting and excising nnet examples. More...
 
void ExampleToPdfPost (const TransitionModel &tmodel, const std::vector< int32 > &silence_phones, std::string criterion, bool drop_frames, bool one_silence_class, const DiscriminativeNnetExample &eg, Posterior *post)
 Given a discriminative training example, this function works out posteriors at the pdf level (note: these are "discriminative-training posteriors" that may be positive or negative. More...
 
void SolvePackingProblem (BaseFloat max_cost, const std::vector< BaseFloat > &costs, std::vector< std::vector< size_t > > *groups)
 This function solves the "packing problem" using the "first fit" algorithm. More...
 
void AppendDiscriminativeExamples (const std::vector< const DiscriminativeNnetExample * > &input, DiscriminativeNnetExample *output)
 Appends the given vector of examples (which must be non-empty) into a single output example (called by CombineExamples, which might be a more convenient interface). More...
 
void CombineDiscriminativeExamples (int32 max_length, const std::vector< DiscriminativeNnetExample > &input, std::vector< DiscriminativeNnetExample > *output)
 This function is used to combine multiple discriminative-training examples (each corresponding to a segment of a lattice), into one. More...
 
bool HasSimpleLabels (const NnetExample &eg, std::vector< int32 > *simple_labels)
 
void FixNnet (const NnetFixConfig &config, Nnet *nnet)
 
int32 IndexOfSoftmaxLayer (const Nnet &nnet)
 If "nnet" has exactly one softmax layer, this function will return its index; otherwise it will return -1. More...
 
void InsertComponents (const Nnet &src_nnet, int32 c, Nnet *dest_nnet)
 Inserts the components of one neural network into a particular place in the other one. More...
 
void ReplaceLastComponents (const Nnet &src_nnet, int32 num_to_remove, Nnet *dest_nnet)
 Removes the last "num_to_remove" components and adds the components from "src_nnet". More...
 
void LimitRankParallel (const NnetLimitRankOpts &opts, Nnet *nnet)
 This function limits the rank of each affine transform in the neural net, by zeroing out the smallest singular values. More...
 
void UnitTestNnet ()
 
NnetGenRandomNnet (int32 input_dim, int32 output_dim)
 This function generates a random neural net, for testing purposes. More...
 
void UnitTestPreconditionDirectionsOnline ()
 
void UnitTestPreconditionDirections ()
 
void PreconditionDirections (const CuMatrixBase< BaseFloat > &R, double lambda, CuMatrixBase< BaseFloat > *P)
 See below for comment. More...
 
void PreconditionDirectionsAlpha (const CuMatrixBase< BaseFloat > &R, double alpha, CuMatrixBase< BaseFloat > *P)
 This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections. More...
 
void PreconditionDirectionsAlphaRescaled (const CuMatrixBase< BaseFloat > &R, double alpha, CuMatrixBase< BaseFloat > *P)
 This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections. More...
 
void GetNnetStats (const NnetStatsConfig &config, const Nnet &nnet, std::vector< NnetStats > *stats)
 
double DoBackpropParallel (const Nnet &nnet, int32 minibatch_size, SequentialNnetExampleReader *example_reader, double *tot_weight, Nnet *nnet_to_update)
 This function is similar to "DoBackprop" in nnet-update.h This function computes the objective function and either updates the model or computes parameter gradients. More...
 
double DoBackpropSingleThreaded (const Nnet &nnet, int32 minibatch_size, const std::vector< NnetExample > &egs, double *tot_weight, Nnet *nnet_to_update)
 
double DoBackpropParallel (const Nnet &nnet, int32 minibatch_size, int32 num_threads, const std::vector< NnetExample > &examples, double *num_frames, Nnet *nnet_to_update)
 This version of DoBackpropParallel takes a vector of examples, and will typically be used to compute the exact gradient. More...
 
double ComputeNnetObjfParallel (const Nnet &nnet, int32 minibatch_size, int32 num_threads, const std::vector< NnetExample > &examples, double *num_frames)
 This is basically to clarify the fact that DoBackpropParallel will also work with nnet_to_update == NULL, and will compute the objf. More...
 
void FormatNnetInput (const Nnet &nnet, const std::vector< NnetExample > &data, Matrix< BaseFloat > *mat)
 Takes the input to the nnet for a minibatch of examples, and formats as a single matrix. More...
 
BaseFloat TotalNnetTrainingWeight (const std::vector< NnetExample > &egs)
 Returns the total weight summed over all the examples... More...
 
double ComputeNnetObjf (const Nnet &nnet, const std::vector< NnetExample > &examples, double *tot_accuracy=NULL)
 Computes objective function over a minibatch. More...
 
double DoBackprop (const Nnet &nnet, const std::vector< NnetExample > &examples, Nnet *nnet_to_update, double *tot_accuracy=NULL)
 This function computes the objective function and either updates the model or adds to parameter gradients. More...
 
double DoBackprop (const Nnet &nnet, const std::vector< NnetExample > &examples, Matrix< BaseFloat > *examples_formatted, Nnet *nnet_to_update, double *tot_accuracy=NULL)
 This version of DoBackprop allows you to separately call FormatNnetInput and provide the result to DoBackprop; this can be useful when using GPUs because the call to FormatNnetInput can be in a separate thread from the one that uses the GPU. More...
 
double ComputeNnetGradient (const Nnet &nnet, const std::vector< NnetExample > &examples, int32 batch_size, Nnet *gradient)
 ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example into batches and calls DoBackprop() on each. More...
 
double ComputeNnetObjf (const Nnet &nnet, const std::vector< NnetExample > &examples, int32 minibatch_size, double *tot_accuracy=NULL)
 This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computation. More...
 
void UnitTestNnetDecodable ()
 
void RescaleNnet (const NnetRescaleConfig &rescale_config, const std::vector< NnetExample > &examples, Nnet *nnet)
 
static BaseFloat ComputeObjfAndGradient (const std::vector< NnetExample > &validation_set, const Vector< double > &log_scale_params, const Nnet &nnet, Vector< double > *gradient)
 
void ShrinkNnet (const NnetShrinkConfig &shrink_config, const std::vector< NnetExample > &validation_set, Nnet *nnet)
 
static Int32Pair MakePair (int32 first, int32 second)
 
int64 TrainNnetSimple (const NnetSimpleTrainerConfig &config, Nnet *nnet, SequentialNnetExampleReader *reader, double *tot_weight=NULL, double *tot_logprob=NULL)
 Train on all the examples it can read from the reader. More...
 
void WidenNnet (const NnetWidenConfig &widen_config, Nnet *nnet)
 This function widens a neural network by increasing the hidden-layer dimensions to the target. More...
 
BaseFloat KlDivergence (const Vector< BaseFloat > &p, const Vector< BaseFloat > &q)
 
void PrintPriorDiagnostics (const Vector< BaseFloat > &old_priors, const Vector< BaseFloat > &new_priors)
 
int32 GetCount (double expected_count)
 
void AverageConstPart (int32 const_feat_dim, DiscriminativeNnetExample *eg)
 
static void ProcessFile (const MatrixBase< BaseFloat > &feats, const Posterior &pdf_post, const std::string &utt_id, int32 left_context, int32 right_context, int32 num_frames, int32 const_feat_dim, int64 *num_frames_written, int64 *num_egs_written, NnetExampleWriter *example_writer)
 
static void ProcessFile (const MatrixBase< BaseFloat > &feats, const Posterior &pdf_post, const std::string &utt_id, const Vector< BaseFloat > &weights, int32 left_context, int32 right_context, int32 const_feat_dim, BaseFloat keep_proportion, BaseFloat weight_threshold, bool use_frame_selection, bool use_frame_weights, int64 *num_frames_written, int64 *num_frames_skipped, NnetExampleWriter *example_writer)
 
void SetMaxChange (BaseFloat max_change, Nnet *nnet)
 
void TransformTrainingExample (const Matrix< BaseFloat > &fmllr_mat, BaseFloat noise_factor, NnetExample *eg)
 
void PerturbTrainingExample (const TpMatrix< BaseFloat > &cholesky, BaseFloat noise_factor, NnetExample *eg)
 
void SetPriors (const TransitionModel &tmodel, const Vector< double > &transition_accs, double prior_floor, AmNnet *am_nnet)
 

Variables

static bool nnet_example_warned_left = false
 
static bool nnet_example_warned_right = false
 

Typedef Documentation

Function Documentation

static void kaldi::nnet2::AddDirection ( const Nnet &  orig_nnet,
const Nnet &  direction,
const VectorBase< BaseFloat > &  scales,
Nnet *  dest 
)
static

Sets "dest" to orig_nnet plus "direction", with each updatable component of "direction" first scaled by the appropriate scale.

Definition at line 52 of file combine-nnet-a.cc.

References Nnet::AddNnet().

Referenced by CombineNnetsA(), and ComputeObjfAndGradient().

55  {
56  *dest = orig_nnet;
57  dest->AddNnet(scales, direction);
58 }
void AppendDiscriminativeExamples ( const std::vector< const DiscriminativeNnetExample * > &  input,
DiscriminativeNnetExample *  output 
)

Appends the given vector of examples (which must be non-empty) into a single output example (called by CombineExamples, which might be a more convenient interface).

When combining examples it directly appends the features, and then adds a "fake" segment to the lattice and alignment in between, padding with transition-ids that are all ones. This is necessary in case the network needs acoustic context, and only because of a kind of limitation in the nnet training code that doesn't support varying 'chunk' sizes within a minibatch.

Will fail if all the input examples don't have the same weight (this will normally be 1.0 anyway), or if the feature dimension (i.e. basic feature dimension plus spk_info dimension) differs between the examples.

Definition at line 891 of file nnet-example-functions.cc.

References DiscriminativeNnetExample::den_lat, VectorBase< Real >::Dim(), rnnlm::i, DiscriminativeNnetExample::input_frames, KALDI_ASSERT, kaldi::kUndefined, DiscriminativeNnetExample::left_context, DiscriminativeNnetExample::num_ali, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), CompactLatticeWeightTpl< WeightType, IntType >::One(), MatrixBase< Real >::Range(), Vector< Real >::Resize(), Matrix< Real >::Resize(), DiscriminativeNnetExample::spk_info, and DiscriminativeNnetExample::weight.

Referenced by CombineDiscriminativeExamples().

893  {
894  KALDI_ASSERT(!input.empty());
895  const DiscriminativeNnetExample &eg0 = *(input[0]);
896 
897  int32 dim = eg0.input_frames.NumCols() + eg0.spk_info.Dim(),
898  left_context = eg0.left_context,
899  num_frames = eg0.num_ali.size(),
900  right_context = eg0.input_frames.NumRows() - num_frames - left_context;
901 
902  int32 tot_frames = eg0.input_frames.NumRows(); // total frames (appended,
903  // with context)
904  for (size_t i = 1; i < input.size(); i++)
905  tot_frames += input[i]->input_frames.NumRows();
906 
907  int32 arbitrary_tid = 1; // arbitrary transition-id that we use to pad the
908  // num_ali and den_lat members between segments
909  // (since they're both the same, and the den-lat in
910  // those parts is linear, they contribute no
911  // derivative to the training).
912 
913  output->den_lat = eg0.den_lat;
914  output->num_ali = eg0.num_ali;
915  output->input_frames.Resize(tot_frames, dim, kUndefined);
916  output->input_frames.Range(0, eg0.input_frames.NumRows(),
917  0, eg0.input_frames.NumCols()).CopyFromMat(eg0.input_frames);
918  if (eg0.spk_info.Dim() != 0) {
919  output->input_frames.Range(0, eg0.input_frames.NumRows(),
920  eg0.input_frames.NumCols(), eg0.spk_info.Dim()).
921  CopyRowsFromVec(eg0.spk_info);
922  }
923 
924  output->num_ali.reserve(tot_frames - left_context - right_context);
925  output->weight = eg0.weight;
926  output->left_context = eg0.left_context;
927  output->spk_info.Resize(0);
928 
929  CompactLattice inter_segment_clat;
930  int32 initial = inter_segment_clat.AddState(); // state 0.
931  inter_segment_clat.SetStart(initial);
932 
933  std::vector<int32> inter_segment_ali(left_context + right_context);
934  std::fill(inter_segment_ali.begin(), inter_segment_ali.end(), arbitrary_tid);
935 
936  CompactLatticeWeight final_weight = CompactLatticeWeight::One();
937  final_weight.SetString(inter_segment_ali);
938  inter_segment_clat.SetFinal(initial, final_weight);
939 
940  int32 feat_offset = eg0.input_frames.NumRows();
941 
942  for (size_t i = 1; i < input.size(); i++) {
943  const DiscriminativeNnetExample &eg_i = *(input[i]);
944 
945  output->input_frames.Range(feat_offset, eg_i.input_frames.NumRows(),
946  0, eg_i.input_frames.NumCols()).CopyFromMat(
947  eg_i.input_frames);
948  if (eg_i.spk_info.Dim() != 0) {
949  output->input_frames.Range(feat_offset, eg_i.input_frames.NumRows(),
950  eg_i.input_frames.NumCols(),
951  eg_i.spk_info.Dim()).CopyRowsFromVec(
952  eg_i.spk_info);
953  KALDI_ASSERT(eg_i.input_frames.NumCols() +
954  eg_i.spk_info.Dim() == dim);
955  }
956 
957  output->num_ali.insert(output->num_ali.end(),
958  inter_segment_ali.begin(), inter_segment_ali.end());
959  output->num_ali.insert(output->num_ali.end(),
960  eg_i.num_ali.begin(), eg_i.num_ali.end());
961  Concat(&(output->den_lat), inter_segment_clat);
962  Concat(&(output->den_lat), eg_i.den_lat);
963  KALDI_ASSERT(output->weight == eg_i.weight);
964  KALDI_ASSERT(output->left_context == eg_i.left_context);
965  feat_offset += eg_i.input_frames.NumRows();
966  }
967  KALDI_ASSERT(feat_offset == tot_frames);
968 }
fst::VectorFst< CompactLatticeArc > CompactLattice
Definition: kaldi-lattice.h:46
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
CompactLatticeWeightTpl< LatticeWeight, int32 > CompactLatticeWeight
void kaldi::nnet2::AverageConstPart ( int32  const_feat_dim,
DiscriminativeNnetExample eg 
)

Definition at line 42 of file nnet-copy-egs-discriminative.cc.

References VectorBase< Real >::AddRowSumMat(), VectorBase< Real >::Dim(), DiscriminativeNnetExample::input_frames, KALDI_ASSERT, MatrixBase< Real >::NumCols(), MatrixBase< Real >::Range(), Vector< Real >::Resize(), and DiscriminativeNnetExample::spk_info.

Referenced by main().

43  {
44  if (eg->spk_info.Dim() != 0) { // already has const part.
45  KALDI_ASSERT(eg->spk_info.Dim() == const_feat_dim);
46  // and nothing to do.
47  } else {
48  int32 dim = eg->input_frames.NumCols(),
49  basic_dim = dim - const_feat_dim;
50  KALDI_ASSERT(const_feat_dim < eg->input_frames.NumCols());
51  Matrix<BaseFloat> mat(eg->input_frames); // copy to non-compressed matrix.
52  eg->input_frames = mat.Range(0, mat.NumRows(), 0, basic_dim);
53  eg->spk_info.Resize(const_feat_dim);
54  eg->spk_info.AddRowSumMat(1.0 / mat.NumRows(),
55  mat.Range(0, mat.NumRows(),
56  basic_dim, const_feat_dim),
57  0.0);
58  }
59 }
void AddRowSumMat(Real alpha, const MatrixBase< Real > &M, Real beta=1.0)
Does *this = alpha * (sum of rows of M) + beta * *this.
void Resize(MatrixIndexT length, MatrixResizeType resize_type=kSetZero)
Set vector to a specified size (can be zero).
Vector< BaseFloat > spk_info
spk_info contains any component of the features that varies slowly or not at all with time (and hence...
Definition: nnet-example.h:171
SubMatrix< Real > Range(const MatrixIndexT row_offset, const MatrixIndexT num_rows, const MatrixIndexT col_offset, const MatrixIndexT num_cols) const
Return a sub-part of matrix.
Definition: kaldi-matrix.h:196
Matrix< BaseFloat > input_frames
The input data– typically with a number of frames [NumRows()] larger than labels.size(), because it includes features to the left and right as needed for the temporal context of the network.
Definition: nnet-example.h:159
MatrixIndexT NumCols() const
Returns number of columns (or zero for emtpy matrix).
Definition: kaldi-matrix.h:61
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
void kaldi::nnet2::BasicDebugTestForSpliceMax ( bool  output = false)

Definition at line 811 of file nnet-component-test.cc.

References SpliceMaxComponent::Backprop(), rnnlm::i, SpliceMaxComponent::Init(), KALDI_LOG, SpliceMaxComponent::OutputDim(), and SpliceMaxComponent::Propagate().

Referenced by main().

811  {
812  int32 C=5,
813  context_len=2,
814  R= 3 + 2*context_len;
815 
817  std::vector<int32> context(2 * context_len + 1);
818  for (int32 i = -1 * context_len; i <= context_len; i++)
819  context[i + context_len] = i;
820  c->Init(C, context);
821  CuMatrix<BaseFloat> in(R, C), in_deriv(R, C);
822  CuMatrix<BaseFloat> out(R, c->OutputDim());
823  ChunkInfo in_info = ChunkInfo(C, 1, 0, R - 1),
824  out_info = ChunkInfo(C, 1, context_len, R - 1 - context_len);
825 
826  in.SetRandn();
827  if (output)
828  KALDI_LOG << in;
829 
830  c->Propagate(in_info, out_info, in, &out);
831 
832  if (output)
833  KALDI_LOG << out;
834 
835  out.Set(5.0);
836 
837  if (output)
838  KALDI_LOG << out;
839 
840  c->Backprop(in_info, out_info, in, in, out, c, &in_deriv);
841 
842  if (output)
843  KALDI_LOG << in_deriv;
844 
845  delete c;
846 }
void Init(int32 dim, std::vector< int32 > context)
ChunkInfo is a class whose purpose is to describe the structure of matrices holding features...
This is as SpliceComponent but outputs the max of any of the inputs (taking the max across time)...
virtual int32 OutputDim() const
Get size of output vectors.
virtual void Backprop(const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component *to_update, CuMatrix< BaseFloat > *in_deriv) const
Perform backward pass propagation of the derivative, and also either update the model (if to_update =...
#define KALDI_LOG
Definition: kaldi-error.h:133
virtual void Propagate(const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const
Perform forward pass propagation Input->Output.
void CombineDiscriminativeExamples ( int32  max_length,
const std::vector< DiscriminativeNnetExample > &  input,
std::vector< DiscriminativeNnetExample > *  output 
)

This function is used to combine multiple discriminative-training examples (each corresponding to a segment of a lattice), into one.

It combines examples into groups such that each group will have a total length (number of rows of the feature matrix) less than or equal to max_length. However, if individual examples are longer than max_length they will still be processed; they will be given their own group.

See also the documentation for AppendDiscriminativeExamples() which gives more details on how we append the examples.

Will fail if all the input examples don't have the same weight (this will normally be 1.0 anyway).

If the spk_info variables are non-empty, it will move them into the features of the output, so the spk_info of the output will be empty but the appropriate speaker vectors will be appended to each row of the features.

Definition at line 970 of file nnet-example-functions.cc.

References AppendDiscriminativeExamples(), rnnlm::i, rnnlm::j, and SolvePackingProblem().

Referenced by main().

973  {
974 
975  std::vector<BaseFloat> costs(input.size());
976  for (size_t i = 0; i < input.size(); i++)
977  costs[i] = static_cast<BaseFloat>(input[i].input_frames.NumRows());
978  std::vector<std::vector<size_t> > groups;
979  SolvePackingProblem(max_length,
980  costs,
981  &groups);
982  output->clear();
983  output->resize(groups.size());
984  for (size_t i = 0; i < groups.size(); i++) {
985  std::vector<const DiscriminativeNnetExample*> group_egs;
986  for (size_t j = 0; j < groups[i].size(); j++) {
987  size_t index = groups[i][j];
988  group_egs.push_back(&(input[index]));
989  }
990  AppendDiscriminativeExamples(group_egs, &((*output)[i]));
991  }
992 }
void AppendDiscriminativeExamples(const std::vector< const DiscriminativeNnetExample * > &input, DiscriminativeNnetExample *output)
Appends the given vector of examples (which must be non-empty) into a single output example (called b...
void SolvePackingProblem(BaseFloat max_cost, const std::vector< BaseFloat > &costs, std::vector< std::vector< size_t > > *groups)
This function solves the "packing problem" using the "first fit" algorithm.
static void kaldi::nnet2::CombineNnets ( const Vector< BaseFloat > &  scale_params,
const std::vector< Nnet > &  nnets,
Nnet *  dest 
)
static

Definition at line 28 of file combine-nnet.cc.

References Nnet::AddNnet(), KALDI_ASSERT, rnnlm::n, kaldi::nnet3::NumUpdatableComponents(), and Nnet::ScaleComponents().

Referenced by CombineNnets(), ComputeObjfAndGradient(), GetInitialModel(), and main().

30  {
31  int32 num_nnets = nnets.size();
32  KALDI_ASSERT(num_nnets >= 1);
33  int32 num_uc = nnets[0].NumUpdatableComponents();
34  KALDI_ASSERT(num_nnets * nnets[0].NumUpdatableComponents());
35 
36 
37  *dest = nnets[0];
38  SubVector<BaseFloat> scale_params0(scale_params, 0, num_uc);
39  dest->ScaleComponents(scale_params0);
40  for (int32 n = 1; n < num_nnets; n++) {
41  SubVector<BaseFloat> scale_params_n(scale_params, n * num_uc, num_uc);
42  dest->AddNnet(scale_params_n, nnets[n]);
43  }
44 }
struct rnnlm::@11::@12 n
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents(const Nnet &dest)
Returns the number of updatable components in the nnet.
Definition: nnet-utils.cc:387
void CombineNnets ( const NnetCombineConfig &  combine_config,
const std::vector< NnetExample > &  validation_set,
const std::vector< Nnet > &  nnets,
Nnet *  nnet_out 
)

Definition at line 193 of file combine-nnet.cc.

References CombineNnets(), ComputeObjfAndGradient(), VectorBase< Real >::CopyFromVec(), MatrixBase< Real >::CopyRowsFromVec(), VectorBase< Real >::Dim(), OptimizeLbfgs< Real >::DoStep(), LbfgsOptions::first_step_impr, GetInitialScaleParams(), OptimizeLbfgs< Real >::GetProposedValue(), OptimizeLbfgs< Real >::GetValue(), rnnlm::i, NnetCombineConfig::initial_impr, KALDI_ASSERT, KALDI_LOG, KALDI_VLOG, LbfgsOptions::m, LbfgsOptions::minimize, NnetCombineConfig::num_bfgs_iters, and NnetCombineConfig::test_gradient.

196  {
197 
198  Vector<double> scale_params;
199 
200  GetInitialScaleParams(combine_config,
201  validation_set,
202  nnets,
203  &scale_params);
204 
205  int32 dim = scale_params.Dim();
206  KALDI_ASSERT(dim > 0);
207  Vector<double> gradient(dim);
208 
209  double objf, initial_objf;
210 
211  LbfgsOptions lbfgs_options;
212  lbfgs_options.minimize = false; // We're maximizing.
213  lbfgs_options.m = dim; // Store the same number of vectors as the dimension
214  // itself, so this is BFGS.
215  lbfgs_options.first_step_impr = combine_config.initial_impr;
216 
217  OptimizeLbfgs<double> lbfgs(scale_params,
218  lbfgs_options);
219 
220  for (int32 i = 0; i < combine_config.num_bfgs_iters; i++) {
221  scale_params.CopyFromVec(lbfgs.GetProposedValue());
222  objf = ComputeObjfAndGradient(validation_set,
223  scale_params,
224  nnets,
225  combine_config.test_gradient,
226  &gradient);
227 
228  KALDI_VLOG(2) << "Iteration " << i << " scale-params = " << scale_params
229  << ", objf = " << objf << ", gradient = " << gradient;
230 
231  if (i == 0) initial_objf = objf;
232 
233  lbfgs.DoStep(objf, gradient);
234  }
235 
236  scale_params.CopyFromVec(lbfgs.GetValue(&objf));
237 
238  Vector<BaseFloat> scale_params_float(scale_params);
239 
240  KALDI_LOG << "Combining nnets, validation objf per frame changed from "
241  << initial_objf << " to " << objf;
242 
243  Matrix<BaseFloat> scale_params_mat(nnets.size(),
244  nnets[0].NumUpdatableComponents());
245  scale_params_mat.CopyRowsFromVec(scale_params_float);
246  KALDI_LOG << "Final scale factors are " << scale_params_mat;
247 
248  CombineNnets(scale_params_float, nnets, nnet_out);
249 }
static void GetInitialScaleParams(const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Vector< double > *scale_params)
Definition: combine-nnet.cc:91
static double ComputeObjfAndGradient(const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const std::vector< Nnet > &nnets, bool debug, Vector< double > *gradient)
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
void CombineNnets(const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)
#define KALDI_LOG
Definition: kaldi-error.h:133
void CombineNnetsA ( const NnetCombineAconfig &  config,
const std::vector< NnetExample > &  validation_set,
const std::vector< Nnet > &  nnets,
Nnet *  nnet_out 
)

Definition at line 102 of file combine-nnet-a.cc.

References AddDirection(), ComputeObjfAndGradient(), VectorBase< Real >::CopyFromVec(), VectorBase< Real >::Dim(), OptimizeLbfgs< Real >::DoStep(), LbfgsOptions::first_step_length, Nnet::GetComponent(), OptimizeLbfgs< Real >::GetProposedValue(), GetUpdateDirection(), OptimizeLbfgs< Real >::GetValue(), rnnlm::i, NnetCombineAconfig::initial_step, rnnlm::j, KALDI_ASSERT, KALDI_LOG, KALDI_VLOG, KALDI_WARN, UpdatableComponent::LearningRate(), LbfgsOptions::m, NnetCombineAconfig::max_learning_rate_factor, NnetCombineAconfig::min_learning_rate, NnetCombineAconfig::min_learning_rate_factor, LbfgsOptions::minimize, NnetCombineAconfig::num_bfgs_iters, Nnet::NumComponents(), kaldi::nnet3::NumUpdatableComponents(), NnetCombineAconfig::overshoot, VectorBase< Real >::Set(), UpdatableComponent::SetLearningRate(), and NnetCombineAconfig::valid_impr_thresh.

Referenced by main().

105  {
106 
107  Nnet direction; // the update direction = avg(nnets[1 ... N]) - nnets[0].
108  GetUpdateDirection(nnets, &direction);
109 
110  Vector<double> scale_params(nnets[0].NumUpdatableComponents()); // initial
111  // scale on "direction".
112 
113  int32 dim = scale_params.Dim();
114  KALDI_ASSERT(dim > 0);
115  Vector<double> gradient(dim);
116 
117  double objf, initial_objf, zero_objf;
118 
119  // Compute objf at zero; we don't actually need this gradient.
120  zero_objf = ComputeObjfAndGradient(validation_set,
121  scale_params,
122  nnets[0],
123  direction,
124  &gradient);
125  KALDI_LOG << "Objective function at old parameters is "
126  << zero_objf;
127 
128  scale_params.Set(1.0); // start optimization from the average of the parameters.
129 
130  LbfgsOptions lbfgs_options;
131  lbfgs_options.minimize = false; // We're maximizing.
132  lbfgs_options.m = dim; // Store the same number of vectors as the dimension
133  // itself, so this is BFGS.
134  lbfgs_options.first_step_length = config.initial_step;
135 
136  OptimizeLbfgs<double> lbfgs(scale_params,
137  lbfgs_options);
138 
139  for (int32 i = 0; i < config.num_bfgs_iters; i++) {
140  scale_params.CopyFromVec(lbfgs.GetProposedValue());
141  objf = ComputeObjfAndGradient(validation_set,
142  scale_params,
143  nnets[0],
144  direction,
145  &gradient);
146 
147  KALDI_VLOG(2) << "Iteration " << i << " scale-params = " << scale_params
148  << ", objf = " << objf << ", gradient = " << gradient;
149 
150  if (i == 0) initial_objf = objf;
151  lbfgs.DoStep(objf, gradient);
152  }
153 
154  scale_params.CopyFromVec(lbfgs.GetValue(&objf));
155 
156  KALDI_LOG << "Combining nnets, after BFGS, validation objf per frame changed from "
157  << zero_objf << " (no change), or " << initial_objf << " (default change), "
158  << " to " << objf << "; scale factors on update direction are "
159  << scale_params;
160 
161  BaseFloat objf_change = objf - zero_objf;
162  KALDI_ASSERT(objf_change >= 0.0); // This is guaranteed by the L-BFGS code.
163 
164  if (objf_change < config.valid_impr_thresh) {
165  // We'll overshoot. To have a smooth transition between the two regimes, if
166  // objf_change is close to valid_impr_thresh we don't overshoot as far.
167  BaseFloat overshoot = config.overshoot,
168  overshoot_max = config.valid_impr_thresh / objf_change; // >= 1.0.
169  if (overshoot_max < overshoot) {
170  KALDI_LOG << "Limiting overshoot from " << overshoot << " to " << overshoot_max
171  << " since the objf-impr " << objf_change << " is close to "
172  << "--valid-impr-thresh=" << config.valid_impr_thresh;
173  overshoot = overshoot_max;
174  }
175  KALDI_ASSERT(overshoot < 2.0 && "--valid-impr-thresh must be < 2.0 or "
176  "it will lead to instability.");
177  scale_params.Scale(overshoot);
178 
179  BaseFloat optimized_objf = objf;
180  objf = ComputeObjfAndGradient(validation_set,
181  scale_params,
182  nnets[0],
183  direction,
184  &gradient);
185 
186  KALDI_LOG << "Combining nnets, after overshooting, validation objf changed "
187  << "to " << objf << ". Note: (zero, start, optimized) objfs were "
188  << zero_objf << ", " << initial_objf << ", " << optimized_objf;
189  if (objf < zero_objf) {
190  // Note: this should not happen according to a quadratic approximation, and we
191  // expect this branch to be taken only rarely if at all.
192  KALDI_WARN << "After overshooting, objf was worse than not updating; not doing the "
193  << "overshoot. ";
194  scale_params.Scale(1.0 / overshoot);
195  }
196  } // Else don't do the "overshoot" stuff.
197 
198  Vector<BaseFloat> scale_params_float(scale_params);
199  // Output to "nnet_out":
200  AddDirection(nnets[0], direction, scale_params_float, nnet_out);
201 
202  // Now update the neural net learning rates.
203  int32 i = 0;
204  for (int32 j = 0; j < nnet_out->NumComponents(); j++) {
205  UpdatableComponent *uc =
206  dynamic_cast<UpdatableComponent*>(&(nnet_out->GetComponent(j)));
207  if (uc != NULL) {
208  BaseFloat step_length = scale_params(i), factor = step_length;
209  // Our basic rule is to update the learning rate by multiplying it
210  // by "step_lenght", but this is subject to certain limits.
211  if (factor < config.min_learning_rate_factor)
212  factor = config.min_learning_rate_factor;
213  if (factor > config.max_learning_rate_factor)
214  factor = config.max_learning_rate_factor;
215  BaseFloat new_learning_rate = factor * uc->LearningRate();
216  if (new_learning_rate < config.min_learning_rate)
217  new_learning_rate = config.min_learning_rate;
218  KALDI_LOG << "For component " << j << ", step length was " << step_length
219  << ", updating learning rate by factor " << factor << ", changing "
220  << "learning rate from " << uc->LearningRate() << " to "
221  << new_learning_rate;
222  uc->SetLearningRate(new_learning_rate);
223  i++;
224  }
225  }
226 }
static void AddDirection(const Nnet &orig_nnet, const Nnet &direction, const VectorBase< BaseFloat > &scales, Nnet *dest)
Sets "dest" to orig_nnet plus "direction", with each updatable component of "direction" first scaled ...
float BaseFloat
Definition: kaldi-types.h:29
static void GetUpdateDirection(const std::vector< Nnet > &nnets, Nnet *direction)
#define KALDI_WARN
Definition: kaldi-error.h:130
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
#define KALDI_LOG
Definition: kaldi-error.h:133
int32 NumUpdatableComponents(const Nnet &dest)
Returns the number of updatable components in the nnet.
Definition: nnet-utils.cc:387
static BaseFloat ComputeObjfAndGradient(const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const Nnet &orig_nnet, const Nnet &direction, Vector< double > *gradient)
void CombineNnetsFast ( const NnetCombineFastConfig &  combine_config,
const std::vector< NnetExample > &  validation_set,
const std::vector< Nnet > &  nnets_in,
Nnet *  nnet_out 
)

Definition at line 430 of file combine-nnet-fast.cc.

Referenced by main().

433  {
434  // Everything happens in the initializer.
435  FastNnetCombiner combiner(combine_config,
436  validation_set,
437  nnets_in,
438  nnet_out);
439 }
double ComputeNnetGradient ( const Nnet &  nnet,
const std::vector< NnetExample > &  examples,
int32  batch_size,
Nnet *  gradient 
)

ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example into batches and calls DoBackprop() on each.

It returns the *average* objective function per frame.

Definition at line 302 of file nnet-update.cc.

References DoBackprop(), rnnlm::i, and Nnet::SetZero().

Referenced by ComputeObjfAndGradient(), and main().

306  {
307  bool treat_as_gradient = true;
308  gradient->SetZero(treat_as_gradient);
309  std::vector<NnetExample> batch;
310  batch.reserve(batch_size);
311  double tot_objf = 0.0;
312  for (int32 start_pos = 0;
313  start_pos < static_cast<int32>(validation_set.size());
314  start_pos += batch_size) {
315  batch.clear();
316  for (int32 i = start_pos;
317  i < std::min(start_pos + batch_size,
318  static_cast<int32>(validation_set.size()));
319  i++) {
320  batch.push_back(validation_set[i]);
321  }
322  tot_objf += DoBackprop(nnet,
323  batch,
324  gradient);
325  }
326  return tot_objf / validation_set.size();
327 }
double DoBackprop(const Nnet &nnet, const std::vector< NnetExample > &examples, Matrix< BaseFloat > *examples_formatted, Nnet *nnet_to_update, double *tot_accuracy)
This version of DoBackprop allows you to separately call FormatNnetInput and provide the result to Do...
Definition: nnet-update.cc:281
double ComputeNnetObjf ( const Nnet &  nnet,
const std::vector< NnetExample > &  examples,
double *  tot_accuracy = NULL 
)

Computes objective function over a minibatch.

Returns the *total* weighted objective function over the minibatch. If tot_accuracy != NULL, it outputs to that pointer the total (weighted) accuracy.

Definition at line 258 of file nnet-update.cc.

References NnetUpdater::ComputeForMinibatch().

Referenced by ComputeNnetObjf(), DoBackprop(), GetInitialModel(), main(), and DoBackpropParallelClass::operator()().

260  {
261  NnetUpdater updater(nnet, NULL);
262  return updater.ComputeForMinibatch(examples, tot_accuracy);
263 }
double ComputeNnetObjf ( const Nnet &  nnet,
const std::vector< NnetExample > &  examples,
int32  minibatch_size,
double *  tot_accuracy = NULL 
)

This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computation.

Returns the *total* (weighted) objective function. If tot_accuracy != NULL, it outputs to that pointer the total (weighted) accuracy.

Definition at line 329 of file nnet-update.cc.

References ComputeNnetObjf(), and rnnlm::i.

333  {
334  double tot_accuracy_tmp;
335  if (tot_accuracy)
336  *tot_accuracy = 0.0;
337  std::vector<NnetExample> batch;
338  batch.reserve(batch_size);
339  double tot_objf = 0.0;
340  for (int32 start_pos = 0;
341  start_pos < static_cast<int32>(validation_set.size());
342  start_pos += batch_size) {
343  batch.clear();
344  for (int32 i = start_pos;
345  i < std::min(start_pos + batch_size,
346  static_cast<int32>(validation_set.size()));
347  i++) {
348  batch.push_back(validation_set[i]);
349  }
350  tot_objf += ComputeNnetObjf(nnet, batch,
351  tot_accuracy != NULL ? &tot_accuracy_tmp : NULL);
352  if (tot_accuracy)
353  *tot_accuracy += tot_accuracy_tmp;
354  }
355  return tot_objf;
356 }
double ComputeNnetObjf(const Nnet &nnet, const std::vector< NnetExample > &validation_set, int32 batch_size, double *tot_accuracy)
This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computatio...
Definition: nnet-update.cc:329
double kaldi::nnet2::ComputeNnetObjfParallel ( const Nnet &  nnet,
int32  minibatch_size,
int32  num_threads,
const std::vector< NnetExample > &  examples,
double *  num_frames 
)
inline

This is basically to clarify the fact that DoBackpropParallel will also work with nnet_to_update == NULL, and will compute the objf.

Both versions of the function will support it, but this version (that takes a vector) is currently the only one we need to do this with.

Definition at line 71 of file nnet-update-parallel.h.

References DoBackpropParallel().

Referenced by FastNnetCombiner::GetInitialModel().

76  {
77  return DoBackpropParallel(nnet, minibatch_size, num_threads,
78  examples, num_frames, NULL);
79 }
double DoBackpropParallel(const Nnet &nnet, int32 minibatch_size, SequentialNnetExampleReader *examples_reader, double *tot_weight, Nnet *nnet_to_update)
This function is similar to "DoBackprop" in nnet-update.h This function computes the objective functi...
static BaseFloat kaldi::nnet2::ComputeObjfAndGradient ( const std::vector< NnetExample > &  validation_set,
const Vector< double > &  log_scale_params,
const Nnet &  nnet,
Vector< double > *  gradient 
)
static

Definition at line 25 of file shrink-nnet.cc.

References VectorBase< Real >::ApplyExp(), ComputeNnetGradient(), VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), Nnet::ScaleComponents(), and Nnet::SetZero().

29  {
30  Vector<BaseFloat> scale_params(log_scale_params);
31  scale_params.ApplyExp();
32  Nnet nnet_scaled(nnet);
33  nnet_scaled.ScaleComponents(scale_params);
34 
35  Nnet nnet_gradient(nnet);
36  bool is_gradient = true;
37  nnet_gradient.SetZero(is_gradient);
38 
39  // note: "ans" is normalized by the total weight of validation frames.
40  int32 batch_size = 1024;
41  BaseFloat ans = ComputeNnetGradient(nnet_scaled,
42  validation_set,
43  batch_size,
44  &nnet_gradient);
45 
46  BaseFloat tot_count = validation_set.size();
47  int32 i = 0; // index into log_scale_params.
48  for (int32 j = 0; j < nnet_scaled.NumComponents(); j++) {
49  const UpdatableComponent *uc =
50  dynamic_cast<const UpdatableComponent*>(&(nnet.GetComponent(j))),
51  *uc_gradient =
52  dynamic_cast<const UpdatableComponent*>(&(nnet_gradient.GetComponent(j)));
53  if (uc != NULL) {
54  BaseFloat dotprod = uc->DotProduct(*uc_gradient) / tot_count;
55  (*gradient)(i) = dotprod * scale_params(i); // gradient w.r.t log of scaling factor.
56  // We multiply by scale_params(i) to take into account d/dx exp(x); "gradient"
57  // is the gradient w.r.t. the log of the scale_params.
58  i++;
59  }
60  }
61  KALDI_ASSERT(i == log_scale_params.Dim());
62  return ans;
63 }
double ComputeNnetGradient(const Nnet &nnet, const std::vector< NnetExample > &validation_set, int32 batch_size, Nnet *gradient)
ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example in...
Definition: nnet-update.cc:302
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
static BaseFloat kaldi::nnet2::ComputeObjfAndGradient ( const std::vector< NnetExample > &  validation_set,
const Vector< double > &  scale_params,
const Nnet &  orig_nnet,
const Nnet &  direction,
Vector< double > *  gradient 
)
static

Definition at line 61 of file combine-nnet-a.cc.

References AddDirection(), ComputeNnetGradient(), VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), and Nnet::SetZero().

Referenced by CombineNnets(), CombineNnetsA(), ComputeObjfAndGradient(), and ShrinkNnet().

66  {
67 
68  Vector<BaseFloat> scale_params_float(scale_params);
69 
70  Nnet nnet_combined;
71  AddDirection(orig_nnet, direction, scale_params_float, &nnet_combined);
72 
73  Nnet nnet_gradient(nnet_combined);
74  bool is_gradient = true;
75  nnet_gradient.SetZero(is_gradient);
76 
77  // note: "ans" is normalized by the total weight of validation frames.
78  int32 batch_size = 1024;
79  BaseFloat ans = ComputeNnetGradient(nnet_combined,
80  validation_set,
81  batch_size,
82  &nnet_gradient);
83 
84  BaseFloat tot_count = validation_set.size();
85  int32 i = 0; // index into scale_params.
86  for (int32 j = 0; j < nnet_combined.NumComponents(); j++) {
87  const UpdatableComponent *uc_direction =
88  dynamic_cast<const UpdatableComponent*>(&(direction.GetComponent(j))),
89  *uc_gradient =
90  dynamic_cast<const UpdatableComponent*>(&(nnet_gradient.GetComponent(j)));
91  if (uc_direction != NULL) {
92  BaseFloat dotprod = uc_direction->DotProduct(*uc_gradient) / tot_count;
93  (*gradient)(i) = dotprod;
94  i++;
95  }
96  }
97  KALDI_ASSERT(i == scale_params.Dim());
98  return ans;
99 }
static void AddDirection(const Nnet &orig_nnet, const Nnet &direction, const VectorBase< BaseFloat > &scales, Nnet *dest)
Sets "dest" to orig_nnet plus "direction", with each updatable component of "direction" first scaled ...
double ComputeNnetGradient(const Nnet &nnet, const std::vector< NnetExample > &validation_set, int32 batch_size, Nnet *gradient)
ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example in...
Definition: nnet-update.cc:302
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
static double kaldi::nnet2::ComputeObjfAndGradient ( const std::vector< NnetExample > &  validation_set,
const Vector< double > &  scale_params,
const std::vector< Nnet > &  nnets,
bool  debug,
Vector< double > *  gradient 
)
static

Definition at line 124 of file combine-nnet.cc.

References CombineNnets(), ComputeNnetGradient(), ComputeObjfAndGradient(), VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, KALDI_LOG, rnnlm::n, Nnet::NumComponents(), and Nnet::SetZero().

129  {
130 
131  Vector<BaseFloat> scale_params_float(scale_params);
132 
133  Nnet nnet_combined;
134  CombineNnets(scale_params_float, nnets, &nnet_combined);
135 
136  Nnet nnet_gradient(nnet_combined);
137  bool is_gradient = true;
138  nnet_gradient.SetZero(is_gradient);
139 
140  // note: "ans" is normalized by the total weight of validation frames.
141  int32 batch_size = 1024;
142  double ans = ComputeNnetGradient(nnet_combined,
143  validation_set,
144  batch_size,
145  &nnet_gradient);
146 
147  double tot_frames = validation_set.size();
148  if (gradient != NULL) {
149  int32 i = 0; // index into scale_params.
150  for (int32 n = 0; n < static_cast<int32>(nnets.size()); n++) {
151  for (int32 j = 0; j < nnet_combined.NumComponents(); j++) {
152  const UpdatableComponent *uc =
153  dynamic_cast<const UpdatableComponent*>(&(nnets[n].GetComponent(j))),
154  *uc_gradient =
155  dynamic_cast<const UpdatableComponent*>(&(nnet_gradient.GetComponent(j)));
156  if (uc != NULL) {
157  double dotprod = uc->DotProduct(*uc_gradient) / tot_frames;
158  (*gradient)(i) = dotprod;
159  i++;
160  }
161  }
162  }
163  KALDI_ASSERT(i == scale_params.Dim());
164  }
165 
166  if (debug) {
167  KALDI_LOG << "Double-checking gradient computation";
168 
169  Vector<BaseFloat> manual_gradient(scale_params.Dim());
170  for (int32 i = 0; i < scale_params.Dim(); i++) {
171  double delta = 1.0e-04, fg = fabs((*gradient)(i));
172  if (fg < 1.0e-07) fg = 1.0e-07;
173  if (fg * delta < 1.0e-05)
174  delta = 1.0e-05 / fg;
175 
176  Vector<double> scale_params_temp(scale_params);
177  scale_params_temp(i) += delta;
178  double new_ans = ComputeObjfAndGradient(validation_set,
179  scale_params_temp,
180  nnets,
181  false,
182  NULL);
183  manual_gradient(i) = (new_ans - ans) / delta;
184  }
185  KALDI_LOG << "Manually computed gradient is " << manual_gradient;
186  KALDI_LOG << "Gradient we computed is " << *gradient;
187  }
188 
189  return ans;
190 }
double ComputeNnetGradient(const Nnet &nnet, const std::vector< NnetExample > &validation_set, int32 batch_size, Nnet *gradient)
ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example in...
Definition: nnet-update.cc:302
struct rnnlm::@11::@12 n
static double ComputeObjfAndGradient(const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const std::vector< Nnet > &nnets, bool debug, Vector< double > *gradient)
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void CombineNnets(const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)
#define KALDI_LOG
Definition: kaldi-error.h:133
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
double DoBackprop ( const Nnet &  nnet,
const std::vector< NnetExample > &  examples,
Nnet *  nnet_to_update,
double *  tot_accuracy = NULL 
)

This function computes the objective function and either updates the model or adds to parameter gradients.

Returns the cross-entropy objective function summed over all samples (normalize this by dividing by TotalNnetTrainingWeight(examples)). It is mostly a wrapper for a class NnetUpdater that's defined in nnet-update.cc, but we don't want to expose that complexity at this level. All these examples will be treated as one minibatch. If tot_accuracy != NULL, it outputs to that pointer the total (weighted) accuracy.

Definition at line 265 of file nnet-update.cc.

References NnetUpdater::ComputeForMinibatch(), ComputeNnetObjf(), Nnet::Info(), and KALDI_LOG.

Referenced by ComputeNnetGradient(), DoBackpropSingleThreaded(), FisherComputationClass::operator()(), DoBackpropParallelClass::operator()(), and TrainNnetSimple().

268  {
269  if (nnet_to_update == NULL)
270  return ComputeNnetObjf(nnet, examples, tot_accuracy);
271  try {
272  NnetUpdater updater(nnet, nnet_to_update);
273  return updater.ComputeForMinibatch(examples, tot_accuracy);
274  } catch (...) {
275  KALDI_LOG << "Error doing backprop, nnet info is: " << nnet.Info();
276  throw;
277  }
278 }
double ComputeNnetObjf(const Nnet &nnet, const std::vector< NnetExample > &validation_set, int32 batch_size, double *tot_accuracy)
This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computatio...
Definition: nnet-update.cc:329
#define KALDI_LOG
Definition: kaldi-error.h:133
double DoBackprop ( const Nnet &  nnet,
const std::vector< NnetExample > &  examples,
Matrix< BaseFloat > *  examples_formatted,
Nnet *  nnet_to_update,
double *  tot_accuracy = NULL 
)

This version of DoBackprop allows you to separately call FormatNnetInput and provide the result to DoBackprop; this can be useful when using GPUs because the call to FormatNnetInput can be in a separate thread from the one that uses the GPU.

"examples_formatted" is really an input, but it's a pointer because internally we call Swap() on it, so we destroy its contents.

Definition at line 281 of file nnet-update.cc.

References NnetUpdater::ComputeForMinibatch(), ComputeNnetObjf(), Nnet::Info(), KALDI_LOG, and KALDI_WARN.

285  {
286  if (nnet_to_update == NULL) {
287  KALDI_WARN << "Was not expecting to reach this code path "
288  << "(wastefully formatting data twice)";
289  return ComputeNnetObjf(nnet, examples, tot_accuracy);
290  } try {
291  NnetUpdater updater(nnet, nnet_to_update);
292  return updater.ComputeForMinibatch(examples,
293  examples_formatted,
294  tot_accuracy);
295  } catch (...) {
296  KALDI_LOG << "Error doing backprop, nnet info is: " << nnet.Info();
297  throw;
298  }
299 }
#define KALDI_WARN
Definition: kaldi-error.h:130
double ComputeNnetObjf(const Nnet &nnet, const std::vector< NnetExample > &validation_set, int32 batch_size, double *tot_accuracy)
This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computatio...
Definition: nnet-update.cc:329
#define KALDI_LOG
Definition: kaldi-error.h:133
double DoBackpropParallel ( const Nnet &  nnet,
int32  minibatch_size,
SequentialNnetExampleReader *  example_reader,
double *  tot_weight,
Nnet *  nnet_to_update 
)

This function is similar to "DoBackprop" in nnet-update.h This function computes the objective function and either updates the model or computes parameter gradients.

It returns the cross-entropy objective function summed over all samples, weighted, and the total weight of the samples (typically the same as the #frames) into total_weight. It is mostly a wrapper for a class NnetUpdater that's defined in nnet-update.cc, but we don't want to expose that complexity at this level. Note: this function If &nnet == nnet_to_update, it assumes we're doing SGD and does something like Hogwild; otherwise it assumes we're computing a gradient and it sums up the gradients. The return value is the total log-prob summed over the #frames. It also outputs the #frames into "num_frames".

Definition at line 147 of file nnet-update-parallel.cc.

References ExamplesRepository::AcceptExamples(), DoBackpropSingleThreaded(), SequentialTableReader< Holder >::Done(), ExamplesRepository::ExamplesDone(), kaldi::g_num_threads, KALDI_LOG, SequentialTableReader< Holder >::Next(), and SequentialTableReader< Holder >::Value().

Referenced by ComputeNnetObjfParallel(), FastNnetCombiner::ComputeObjfAndGradient(), and main().

151  {
152 #if HAVE_CUDA == 1
153  // Our GPU code won't work with multithreading; we do this
154  // to enable it to work with this code in the single-threaded
155  // case.
156  if (CuDevice::Instantiate().Enabled())
157  return DoBackpropSingleThreaded(nnet, minibatch_size, examples_reader,
158  tot_weight, nnet_to_update);
159 #endif
160 
161  ExamplesRepository repository; // handles parallel programming issues regarding
162  // the "examples" of data.
163  double tot_log_prob = 0.0;
164  *tot_weight = 0.0;
165 
166  // This function assumes you want the exact gradient, if
167  // nnet_to_update != &nnet.
168  const bool store_separate_gradients = (nnet_to_update != &nnet);
169 
170  DoBackpropParallelClass c(nnet, &repository, tot_weight,
171  &tot_log_prob, nnet_to_update,
172  store_separate_gradients);
173 
174  {
175  // The initialization of the following class spawns the threads that
176  // process the examples. They get re-joined in its destructor.
177  MultiThreader<DoBackpropParallelClass> m(g_num_threads, c);
178 
179  std::vector<NnetExample> examples;
180  for (; !examples_reader->Done(); examples_reader->Next()) {
181  examples.push_back(examples_reader->Value());
182  if (examples.size() == minibatch_size)
183  repository.AcceptExamples(&examples);
184  }
185  if (!examples.empty()) // partial minibatch.
186  repository.AcceptExamples(&examples);
187  // Here, the destructor of "m" re-joins the threads, and
188  // does the summing of the gradients if we're doing gradient
189  // computation (i.e. &nnet != nnet_to_update). This gets
190  // done in the destructors of the objects of type
191  // DoBackpropParallelClass.
192  repository.ExamplesDone();
193  }
194  KALDI_LOG << "Did backprop on " << *tot_weight << " examples, average log-prob "
195  << "per frame is " << (tot_log_prob / *tot_weight);
196  KALDI_LOG << "[this line is to be parsed by a script:] log-prob-per-frame="
197  << (tot_log_prob / *tot_weight);
198  return tot_log_prob;
199 }
double DoBackpropSingleThreaded(const Nnet &nnet, int32 minibatch_size, const std::vector< NnetExample > &egs, double *tot_weight, Nnet *nnet_to_update)
int32 g_num_threads
Definition: kaldi-thread.cc:25
#define KALDI_LOG
Definition: kaldi-error.h:133
double DoBackpropParallel ( const Nnet &  nnet,
int32  minibatch_size,
int32  num_threads,
const std::vector< NnetExample > &  examples,
double *  num_frames,
Nnet *  nnet_to_update 
)

This version of DoBackpropParallel takes a vector of examples, and will typically be used to compute the exact gradient.

Definition at line 221 of file nnet-update-parallel.cc.

References ExamplesRepository::AcceptExamples(), DoBackpropSingleThreaded(), ExamplesRepository::ExamplesDone(), and KALDI_VLOG.

226  {
227  if (num_threads == 1) // support GPUs: special case for 1 thread.
228  return DoBackpropSingleThreaded(nnet, minibatch_size, egs,
229  tot_weight, nnet_to_update);
230 
231  ExamplesRepository repository; // handles parallel programming issues regarding
232  // the "examples" of data.
233  double tot_log_prob = 0.0;
234  *tot_weight = 0;
235  const bool store_separate_gradients = (nnet_to_update != &nnet);
236 
237  DoBackpropParallelClass c(nnet, &repository, tot_weight,
238  &tot_log_prob, nnet_to_update,
239  store_separate_gradients);
240 
241  {
242  // The initialization of the following class spawns the threads that
243  // process the examples. They get re-joined in its destructor.
244  MultiThreader<DoBackpropParallelClass> m(num_threads, c);
245 
246  int32 num_egs = egs.size();
247  for (int32 offset = 0; offset < num_egs; offset += minibatch_size) {
248  int32 this_minibatch_size = std::min(minibatch_size, num_egs - offset);
249 
250  // We waste a little time copying the examples here, but it's very minor.
251  std::vector<NnetExample> examples(egs.begin() + offset,
252  egs.begin() + offset + this_minibatch_size);
253 
254  repository.AcceptExamples(&examples);
255  }
256 
257  // Here, the destructor of "m" re-joins the threads, and
258  // does the summing of the gradients if we're doing gradient
259  // computation (i.e. &nnet != nnet_to_update). This gets
260  // done in the destructors of the objects of type
261  // DoBackpropParallelClass.
262  repository.ExamplesDone();
263  }
264  KALDI_VLOG(2) << "Did backprop on " << *tot_weight << " examples, average log-prob "
265  << "per frame is " << (tot_log_prob / *tot_weight);
266  return tot_log_prob;
267 }
double DoBackpropSingleThreaded(const Nnet &nnet, int32 minibatch_size, const std::vector< NnetExample > &egs, double *tot_weight, Nnet *nnet_to_update)
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
double kaldi::nnet2::DoBackpropSingleThreaded ( const Nnet &  nnet,
int32  minibatch_size,
const std::vector< NnetExample > &  egs,
double *  tot_weight,
Nnet *  nnet_to_update 
)

Definition at line 202 of file nnet-update-parallel.cc.

References DoBackprop(), rnnlm::i, and TotalNnetTrainingWeight().

Referenced by DoBackpropParallel().

206  {
207  double ans = 0.0;
208  *tot_weight = TotalNnetTrainingWeight(egs);
209  for (size_t i = 0; i < egs.size(); i += minibatch_size) {
210  std::vector<NnetExample>::const_iterator end_iter =
211  (i + minibatch_size > egs.size() ? egs.end() :
212  egs.begin() + i + minibatch_size);
213  std::vector<NnetExample> this_egs(egs.begin() + i,
214  end_iter);
215  ans += DoBackprop(nnet, this_egs, nnet_to_update);
216  }
217  return ans;
218 }
double DoBackprop(const Nnet &nnet, const std::vector< NnetExample > &examples, Nnet *nnet_to_update, double *tot_accuracy)
This function computes the objective function and either updates the model or adds to parameter gradi...
Definition: nnet-update.cc:265
BaseFloat TotalNnetTrainingWeight(const std::vector< NnetExample > &egs)
Returns the total weight summed over all the examples...
Definition: nnet-update.cc:248
void ExampleToPdfPost ( const TransitionModel &  tmodel,
const std::vector< int32 > &  silence_phones,
std::string  criterion,
bool  drop_frames,
bool  one_silence_class,
const DiscriminativeNnetExample &  eg,
Posterior *  post 
)

Given a discriminative training example, this function works out posteriors at the pdf level (note: these are "discriminative-training posteriors" that may be positive or negative.

The denominator lattice "den_lat" in the example "eg" should already have had acoustic-rescoring done so that its acoustic probs are up to date, and any acoustic scaling should already have been applied.

"criterion" may be "mmi" or "mpfe" or "smbr". If criterion is "mmi", "drop_frames" means we don't include derivatives for frames where the numerator pdf is not in the denominator lattice.

if "one_silence_class" is true you can get a newer behavior for MPE/SMBR which will tend to reduce insertions.

"silence_phones" is a list of silence phones (this is only relevant for mpfe or smbr, if we want to treat silence specially).

Definition at line 838 of file nnet-example-functions.cc.

References fst::ConvertLattice(), kaldi::ConvertPosteriorToPdfs(), DiscriminativeNnetExample::den_lat, KALDI_ASSERT, kaldi::LatticeForwardBackwardMmi(), kaldi::LatticeForwardBackwardMpeVariants(), DiscriminativeNnetExample::num_ali, kaldi::ScalePosterior(), and DiscriminativeNnetExample::weight.

Referenced by UpdateHash().

845  {
846  KALDI_ASSERT(criterion == "mpfe" || criterion == "smbr" || criterion == "mmi");
847 
848  Lattice lat;
849  ConvertLattice(eg.den_lat, &lat);
850  TopSort(&lat);
851  if (criterion == "mpfe" || criterion == "smbr") {
852  Posterior tid_post;
853  LatticeForwardBackwardMpeVariants(tmodel, silence_phones, lat, eg.num_ali,
854  criterion, one_silence_class, &tid_post);
855 
856  ConvertPosteriorToPdfs(tmodel, tid_post, post);
857  } else {
858  bool convert_to_pdf_ids = true, cancel = true;
859  LatticeForwardBackwardMmi(tmodel, lat, eg.num_ali,
860  drop_frames, convert_to_pdf_ids, cancel,
861  post);
862  }
863  ScalePosterior(eg.weight, post);
864 }
std::vector< std::vector< std::pair< int32, BaseFloat > > > Posterior
Posterior is a typedef for storing acoustic-state (actually, transition-id) posteriors over an uttera...
Definition: posterior.h:43
BaseFloat LatticeForwardBackwardMmi(const TransitionModel &tmodel, const Lattice &lat, const std::vector< int32 > &num_ali, bool drop_frames, bool convert_to_pdf_ids, bool cancel, Posterior *post)
This function can be used to compute posteriors for MMI, with a positive contribution for the numerat...
void ConvertLattice(const ExpandedFst< ArcTpl< Weight > > &ifst, MutableFst< ArcTpl< CompactLatticeWeightTpl< Weight, Int > > > *ofst, bool invert)
Convert lattice from a normal FST to a CompactLattice FST.
fst::VectorFst< LatticeArc > Lattice
Definition: kaldi-lattice.h:44
BaseFloat LatticeForwardBackwardMpeVariants(const TransitionModel &trans, const std::vector< int32 > &silence_phones, const Lattice &lat, const std::vector< int32 > &num_ali, std::string criterion, bool one_silence_class, Posterior *post)
This function implements either the MPFE (minimum phone frame error) or SMBR (state-level minimum bay...
void ScalePosterior(BaseFloat scale, Posterior *post)
Scales the BaseFloat (weight) element in the posterior entries.
Definition: posterior.cc:218
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void ConvertPosteriorToPdfs(const TransitionModel &tmodel, const Posterior &post_in, Posterior *post_out)
Converts a posterior over transition-ids to be a posterior over pdf-ids.
Definition: posterior.cc:322
void ExciseDiscriminativeExample ( const SplitDiscriminativeExampleConfig &  config,
const TransitionModel &  tmodel,
const DiscriminativeNnetExample &  eg,
std::vector< DiscriminativeNnetExample > *  egs_out,
SplitExampleStats *  stats_out 
)

Remove unnecessary frames from discriminative training example.

The output egs_out will be of size zero or one (usually one) after being called.

Definition at line 775 of file nnet-example-functions.cc.

References DiscriminativeExampleSplitter::Excise().

Referenced by main().

780  {
781  DiscriminativeExampleSplitter splitter(config, tmodel, eg, egs_out);
782  splitter.Excise(stats_out);
783 }
static void kaldi::nnet2::ExpectOneOrTwoTokens ( std::istream &  is,
bool  binary,
const std::string &  token1,
const std::string &  token2 
)
static

Definition at line 135 of file nnet-component.cc.

References kaldi::ExpectToken(), KALDI_ASSERT, KALDI_ERR, and kaldi::ReadToken().

Referenced by NonlinearComponent::Read(), MaxoutComponent::Read(), MaxpoolingComponent::Read(), PnormComponent::Read(), PowerComponent::Read(), ScaleComponent::Read(), AffineComponent::Read(), AffineComponentPreconditioned::Read(), AffineComponentPreconditionedOnline::Read(), SpliceComponent::Read(), SpliceMaxComponent::Read(), BlockAffineComponent::Read(), BlockAffineComponentPreconditioned::Read(), SumGroupComponent::Read(), PermuteComponent::Read(), DctComponent::Read(), FixedLinearComponent::Read(), FixedAffineComponent::Read(), FixedScaleComponent::Read(), FixedBiasComponent::Read(), DropoutComponent::Read(), AdditiveNoiseComponent::Read(), and Convolutional1dComponent::Read().

137  {
138  KALDI_ASSERT(token1 != token2);
139  std::string temp;
140  ReadToken(is, binary, &temp);
141  if (temp == token1) {
142  ExpectToken(is, binary, token2);
143  } else {
144  if (temp != token2) {
145  KALDI_ERR << "Expecting token " << token1 << " or " << token2
146  << " but got " << temp;
147  }
148  }
149 }
void ReadToken(std::istream &is, bool binary, std::string *str)
ReadToken gets the next token and puts it in str (exception on failure).
Definition: io-funcs.cc:154
void ExpectToken(std::istream &is, bool binary, const char *token)
ExpectToken tries to read in the given token, and throws an exception on failure. ...
Definition: io-funcs.cc:188
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void FixNnet ( const NnetFixConfig &  config,
Nnet *  nnet 
)

Definition at line 31 of file nnet-fix.cc.

References AffineComponent::BiasParams(), count, NonlinearComponent::Count(), rnnlm::d, NonlinearComponent::DerivSum(), Nnet::GetComponent(), NonlinearComponent::InputDim(), KALDI_ASSERT, KALDI_LOG, KALDI_WARN, AffineComponent::LinearParams(), NnetFixConfig::max_average_deriv, NnetFixConfig::min_average_deriv, Nnet::NumComponents(), NnetFixConfig::parameter_factor, NnetFixConfig::relu_bias_change, and AffineComponent::SetParams().

Referenced by main().

31  {
32  for (int32 c = 0; c + 1 < nnet->NumComponents(); c++) {
33  AffineComponent *ac = dynamic_cast<AffineComponent*>(
34  &(nnet->GetComponent(c)));
35  NonlinearComponent *nc = dynamic_cast<NonlinearComponent*>(
36  &(nnet->GetComponent(c + 1)));
37  if (ac == NULL || nc == NULL) continue;
38  // We only want to process this if it's of type SigmoidComponent
39  // or TanhComponent.
40  BaseFloat max_deriv; // The maximum derivative of this nonlinearity.
41  bool is_relu = false;
42  {
43  SigmoidComponent *sc = dynamic_cast<SigmoidComponent*>(nc);
44  TanhComponent *tc = dynamic_cast<TanhComponent*>(nc);
45  RectifiedLinearComponent *rc = dynamic_cast<RectifiedLinearComponent*>(nc);
46  if (sc != NULL) max_deriv = 0.25;
47  else if (tc != NULL) max_deriv = 1.0;
48  else if (rc != NULL) { max_deriv = 1.0; is_relu = true; }
49  else continue; // E.g. SoftmaxComponent; we don't handle this.
50  }
51  double count = nc->Count();
52  Vector<double> deriv_sum (nc->DerivSum());
53  if (count == 0.0 || deriv_sum.Dim() == 0) {
54  KALDI_WARN << "Cannot fix neural net because no statistics are stored.";
55  continue;
56  }
57  Vector<BaseFloat> bias_params(ac->BiasParams());
58  Matrix<BaseFloat> linear_params(ac->LinearParams());
59  int32 dim = nc->InputDim(), num_small_deriv = 0, num_large_deriv = 0;
60  for (int32 d = 0; d < dim; d++) {
61  // deriv ratio is the ratio of the computed average derivative to the
62  // maximum derivative of that nonlinear function.
63  BaseFloat deriv_ratio = deriv_sum(d) / (count * max_deriv);
64  KALDI_ASSERT(deriv_ratio >= 0.0 && deriv_ratio < 1.01); // Or there is an
65  // error in the
66  // math.
67  if (deriv_ratio < config.min_average_deriv) {
68  // derivative is too small, meaning we've gone off into the "flat part"
69  // of the sigmoid (or for ReLU, we're always-off).
70  if (is_relu) {
71  bias_params(d) += config.relu_bias_change;
72  } else {
73  BaseFloat parameter_factor = std::min(config.min_average_deriv /
74  deriv_ratio,
75  config.parameter_factor);
76  // we need to reduce the parameters, so multiply by 1/parameter factor.
77  bias_params(d) *= 1.0 / parameter_factor;
78  linear_params.Row(d).Scale(1.0 / parameter_factor);
79  }
80  num_small_deriv++;
81  } else if (deriv_ratio > config.max_average_deriv) {
82  // derivative is too large, meaning we're only in the linear part of the
83  // sigmoid, in the middle. (or for ReLU, we're always-on.
84  if (is_relu) {
85  bias_params(d) -= config.relu_bias_change;
86  } else {
87  BaseFloat parameter_factor = std::min(deriv_ratio / config.max_average_deriv,
88  config.parameter_factor);
89  // we need to increase the factors, so multiply by parameter_factor.
90  bias_params(d) *= parameter_factor;
91  linear_params.Row(d).Scale(parameter_factor);
92  }
93  num_large_deriv++;
94  }
95  }
96  if (is_relu) {
97  KALDI_LOG << "For layer " << c << " (ReLU units), increased bias for "
98  << num_small_deriv << " indexes and decreased it for "
99  << num_large_deriv << ", out of a total of " << dim;
100  } else {
101  KALDI_LOG << "For layer " << c << ", decreased parameters for "
102  << num_small_deriv << " indexes, and increased them for "
103  << num_large_deriv << " out of a total of " << dim;
104  }
105  ac->SetParams(bias_params, linear_params);
106  }
107 }
const size_t count
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_WARN
Definition: kaldi-error.h:130
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_LOG
Definition: kaldi-error.h:133
void FormatNnetInput ( const Nnet &  nnet,
const std::vector< NnetExample > &  data,
Matrix< BaseFloat > *  mat 
)

Takes the input to the nnet for a minibatch of examples, and formats as a single matrix.

data.size() must be > 0. Note: you will probably want to copy this to CuMatrix after you call this function. The num-rows of the output will, at exit, equal (1 + nnet.LeftContext() + nnet.RightContext()) * data.size(). The nnet is only needed so we can call LeftContext(), RightContext() and InputDim() on it.

Definition at line 207 of file nnet-update.cc.

References MatrixBase< Real >::CopyFromMat(), MatrixBase< Real >::CopyRowsFromVec(), Nnet::InputDim(), KALDI_ASSERT, kaldi::kUndefined, Nnet::LeftContext(), Matrix< Real >::Resize(), and Nnet::RightContext().

Referenced by NnetUpdater::FormatInput(), and NnetExampleBackgroundReader::ReadExamples().

209  {
210  KALDI_ASSERT(data.size() > 0);
211  int32 num_splice = 1 + nnet.RightContext() + nnet.LeftContext();
212  KALDI_ASSERT(data[0].input_frames.NumRows() >= num_splice);
213 
214  int32 feat_dim = data[0].input_frames.NumCols(),
215  spk_dim = data[0].spk_info.Dim(),
216  tot_dim = feat_dim + spk_dim; // we append these at the neural net
217  // input... note, spk_dim might be 0.
218  KALDI_ASSERT(tot_dim == nnet.InputDim());
219  KALDI_ASSERT(data[0].left_context >= nnet.LeftContext());
220  int32 ignore_frames = data[0].left_context - nnet.LeftContext(); // If
221  // the NnetExample has more left-context than we need, ignore some.
222  // this may happen in settings where we increase the amount of context during
223  // training, e.g. by adding layers that require more context.
224 
225  int32 num_chunks = data.size();
226 
227  input_mat->Resize(num_splice * num_chunks,
228  tot_dim, kUndefined);
229 
230  for (int32 chunk = 0; chunk < num_chunks; chunk++) {
231  SubMatrix<BaseFloat> dest(*input_mat,
232  chunk * num_splice, num_splice,
233  0, feat_dim);
234 
235  Matrix<BaseFloat> full_src(data[chunk].input_frames);
236  SubMatrix<BaseFloat> src(full_src, ignore_frames, num_splice, 0, feat_dim);
237 
238  dest.CopyFromMat(src);
239  if (spk_dim != 0) {
240  SubMatrix<BaseFloat> spk_dest(*input_mat,
241  chunk * num_splice, num_splice,
242  feat_dim, spk_dim);
243  spk_dest.CopyRowsFromVec(data[chunk].spk_info);
244  }
245  }
246 }
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
Nnet * GenRandomNnet ( int32  input_dim,
int32  output_dim 
)

This function generates a random neural net, for testing purposes.

It will contain a random number of SigmoidComponent, AffineComponent and SpliceComponent, followed by a final AffineComponent and SoftmaxComponent. The parameters will all be randomly initialized.

Definition at line 772 of file nnet-nnet.cc.

References rnnlm::i, Nnet::Init(), AffineComponent::Init(), and SpliceComponent::Init().

Referenced by UnitTestAmNnet(), UnitTestNnet(), UnitTestNnetCompute(), UnitTestNnetComputeChunked(), and UnitTestNnetDecodable().

773  {
774  std::vector<Component*> components;
775  int32 cur_dim = input_dim;
776  // have up to 10 layers before the final one.
777  for (size_t i = 0; i < 10; i++) {
778  if (rand() % 2 == 0) {
779  // add an affine component.
780  int32 next_dim = 50 + rand() % 100;
781  BaseFloat learning_rate = 0.0001, param_stddev = 0.001,
782  bias_stddev = 0.1;
783  AffineComponent *component = new AffineComponent();
784  component->Init(learning_rate, cur_dim, next_dim,
785  param_stddev, bias_stddev);
786  components.push_back(component);
787  cur_dim = next_dim;
788  } else if (rand() % 2 == 0) {
789  components.push_back(new SigmoidComponent(cur_dim));
790  } else if (rand() % 2 == 0 && cur_dim < 200) {
791  SpliceComponent *component = new SpliceComponent();
792  std::vector<int32> context;
793  while (true) {
794  context.clear();
795  for (int32 i = -3; i <= 3; i++) {
796  if (rand() % 3 == 0)
797  context.push_back(i);
798  }
799  if (!context.empty() && context.front() <= 0 &&
800  context.back() >= 0)
801  break;
802  }
803  component->Init(cur_dim, context);
804  components.push_back(component);
805  cur_dim = cur_dim * context.size();
806  } else {
807  break;
808  }
809  }
810 
811  {
812  AffineComponent *component = new AffineComponent();
813  BaseFloat learning_rate = 0.0001, param_stddev = 0.001,
814  bias_stddev = 0.1;
815  component->Init(learning_rate, cur_dim, output_dim,
816  param_stddev, bias_stddev);
817  components.push_back(component);
818  cur_dim = output_dim;
819  }
820 
821  components.push_back(new SoftmaxComponent(cur_dim));
822 
823  Nnet *ans = new Nnet();
824  ans->Init(&components);
825  return ans;
826 }
float BaseFloat
Definition: kaldi-types.h:29
int32 GetCount ( double  expected_count)

Definition at line 31 of file nnet-copy-egs-discriminative.cc.

References KALDI_ASSERT, and kaldi::WithProb().

Referenced by main(), and ProcessFile().

31  {
32  KALDI_ASSERT(expected_count >= 0.0);
33  int32 ans = 0;
34  while (expected_count > 1.0) {
35  ans++;
36  expected_count--;
37  }
38  if (WithProb(expected_count))
39  ans++;
40  return ans;
41 }
bool WithProb(BaseFloat prob, struct RandomState *state)
Definition: kaldi-math.cc:71
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
static int32 kaldi::nnet2::GetInitialModel ( const std::vector< NnetExample > &  validation_set,
const std::vector< Nnet > &  nnets 
)
static

Returns an integer saying which model to use: either 0 ...

num-models - 1 for the best individual model, or (#models) for the average of all of them.

Definition at line 49 of file combine-nnet.cc.

References CombineNnets(), ComputeNnetObjf(), KALDI_ASSERT, KALDI_LOG, rnnlm::n, and VectorBase< Real >::Set().

Referenced by GetInitialScaleParams().

51  {
52  int32 minibatch_size = 1024;
53  int32 num_nnets = static_cast<int32>(nnets.size());
54  KALDI_ASSERT(!nnets.empty());
55  BaseFloat tot_frames = validation_set.size();
56  int32 best_n = -1;
57  BaseFloat best_objf;
58  Vector<BaseFloat> objfs(nnets.size());
59  for (int32 n = 0; n < num_nnets; n++) {
60  BaseFloat objf = ComputeNnetObjf(nnets[n], validation_set,
61  minibatch_size) / tot_frames;
62 
63  if (n == 0 || objf > best_objf) {
64  best_objf = objf;
65  best_n = n;
66  }
67  objfs(n) = objf;
68  }
69  KALDI_LOG << "Objective functions for the source neural nets are " << objfs;
70 
71  int32 num_uc = nnets[0].NumUpdatableComponents();
72 
73  { // Now try a version where all the neural nets have the same weight.
74  Vector<BaseFloat> scale_params(num_uc * num_nnets);
75  scale_params.Set(1.0 / num_nnets);
76  Nnet average_nnet;
77  CombineNnets(scale_params, nnets, &average_nnet);
78  BaseFloat objf = ComputeNnetObjf(average_nnet, validation_set,
79  minibatch_size) / tot_frames;
80  KALDI_LOG << "Objf with all neural nets averaged is " << objf;
81  if (objf > best_objf) {
82  return num_nnets;
83  } else {
84  return best_n;
85  }
86  }
87 }
double ComputeNnetObjf(const Nnet &nnet, const std::vector< NnetExample > &examples, double *tot_accuracy)
Computes objective function over a minibatch.
Definition: nnet-update.cc:258
float BaseFloat
Definition: kaldi-types.h:29
struct rnnlm::@11::@12 n
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void CombineNnets(const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)
#define KALDI_LOG
Definition: kaldi-error.h:133
static void kaldi::nnet2::GetInitialScaleParams ( const NnetCombineConfig &  combine_config,
const std::vector< NnetExample > &  validation_set,
const std::vector< Nnet > &  nnets,
Vector< double > *  scale_params 
)
static

Definition at line 91 of file combine-nnet.cc.

References GetInitialModel(), NnetCombineConfig::initial_model, KALDI_ASSERT, KALDI_LOG, Vector< Real >::Resize(), and VectorBase< Real >::Set().

Referenced by CombineNnets().

95  {
96 
97  int32 initial_model = combine_config.initial_model,
98  num_nnets = static_cast<int32>(nnets.size());
99  if (initial_model < 0 || initial_model > num_nnets)
100  initial_model = GetInitialModel(validation_set, nnets);
101 
102  KALDI_ASSERT(initial_model >= 0 && initial_model <= num_nnets);
103  int32 num_uc = nnets[0].NumUpdatableComponents();
104 
105  scale_params->Resize(num_uc * num_nnets);
106  if (initial_model < num_nnets) {
107  KALDI_LOG << "Initializing with neural net with index " << initial_model;
108  // At this point we're using the best of the individual neural nets.
109  scale_params->Set(0.0);
110 
111  // Set the block of parameters corresponding to the "best" of the
112  // source neural nets to
113  SubVector<double> best_block(*scale_params, num_uc * initial_model, num_uc);
114  best_block.Set(1.0);
115  } else { // initial_model == num_nnets
116  KALDI_LOG << "Initializing with all neural nets averaged.";
117  scale_params->Set(1.0 / num_nnets);
118  }
119 }
static int32 GetInitialModel(const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets)
Returns an integer saying which model to use: either 0 ...
Definition: combine-nnet.cc:49
void Resize(MatrixIndexT length, MatrixResizeType resize_type=kSetZero)
Set vector to a specified size (can be zero).
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void Set(Real f)
Set all members of a vector to a specified value.
#define KALDI_LOG
Definition: kaldi-error.h:133
void GetNnetStats ( const NnetStatsConfig &  config,
const Nnet &  nnet,
std::vector< NnetStats > *  stats 
)

Definition at line 99 of file nnet-stats.cc.

References NnetStatsConfig::bucket_width, Nnet::GetComponent(), KALDI_ASSERT, NnetStats::NnetStats(), and Nnet::NumComponents().

Referenced by main().

101  {
102  KALDI_ASSERT(stats->size() == 0);
103  for (int32 c = 0; c + 1 < nnet.NumComponents(); c++) {
104  const AffineComponent *ac = dynamic_cast<const AffineComponent*>(
105  &(nnet.GetComponent(c)));
106  if (ac == NULL) continue;
107  const NonlinearComponent *nc = dynamic_cast<const NonlinearComponent*>(
108  &(nnet.GetComponent(c + 1)));
109  if (nc == NULL) continue;
110  // exclude softmax.
111  const SoftmaxComponent *sc = dynamic_cast<const SoftmaxComponent*>(
112  &(nnet.GetComponent(c + 1)));
113  if (sc != NULL) continue;
114  stats->push_back(NnetStats(c, config.bucket_width));
115  stats->back().AddStatsFromNnet(nnet);
116  }
117 }
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
static void kaldi::nnet2::GetUpdateDirection ( const std::vector< Nnet > &  nnets,
Nnet *  direction 
)
static

Definition at line 31 of file combine-nnet-a.cc.

References Nnet::AddNnet(), KALDI_ASSERT, rnnlm::n, kaldi::nnet3::NumUpdatableComponents(), Nnet::ScaleComponents(), and VectorBase< Real >::Set().

Referenced by CombineNnetsA().

32  {
33  KALDI_ASSERT(nnets.size() > 1);
34  int32 num_new_nnets = nnets.size() - 1;
35  Vector<BaseFloat> scales(nnets[0].NumUpdatableComponents());
36 
37  scales.Set(1.0 / num_new_nnets);
38 
39  *direction = nnets[1];
40  direction->ScaleComponents(scales); // first of the new nnets.
41  for (int32 n = 2; n < 1 + num_new_nnets; n++)
42  direction->AddNnet(scales, nnets[n]);
43  // now "direction" is the average of the new nnets. Subtract
44  // the old nnet's parameters.
45  scales.Set(-1.0);
46  direction->AddNnet(scales, nnets[0]);
47 }
struct rnnlm::@11::@12 n
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
int32 NumUpdatableComponents(const Nnet &dest)
Returns the number of updatable components in the nnet.
Definition: nnet-utils.cc:387
static void kaldi::nnet2::GiveNnetCorrectTopology ( Nnet *  nnet,
AffineComponent **  affine_component,
SoftmaxComponent **  softmax_component,
SumGroupComponent **  sum_group_component 
)
static

This function makes sure the neural net ends with a SumGroupComponent.

If it doesn't, it adds one (with a single mixture/matrix corresponding to each output element.) [Before doing so, it makes sure that the last layer is a SoftmaxLayer, which is what we expect. You can remove this check if there is some use-case that makes sense where the type of the previous layer is different.

Definition at line 37 of file mixup-nnet.cc.

References Nnet::Append(), Nnet::GetComponent(), KALDI_ASSERT, KALDI_ERR, KALDI_LOG, Nnet::NumComponents(), Component::OutputDim(), and Component::Type().

Referenced by MixupNnet().

40  {
41  int32 nc = nnet->NumComponents();
42  KALDI_ASSERT(nc > 0);
43  Component* component = &(nnet->GetComponent(nc - 1));
44  if ((*sum_group_component =
45  dynamic_cast<SumGroupComponent*>(component)) == NULL) {
46  KALDI_LOG << "Adding SumGroupComponent to neural net.";
47  int32 dim = component->OutputDim();
48  // Give it the same learning rate as the first updatable layer we have.
49  std::vector<int32> sizes(dim, 1); // a vector of all ones, of dimension "dim".
50 
51  *sum_group_component = new SumGroupComponent();
52  (*sum_group_component)->Init(sizes);
53  nnet->Append(*sum_group_component);
54  nc++;
55  }
56  component = &(nnet->GetComponent(nc - 2));
57  if ((*softmax_component = dynamic_cast<SoftmaxComponent*>(component)) == NULL)
58  KALDI_ERR << "Neural net has wrong topology: expected second-to-last "
59  << "component to be SoftmaxComponent, type is "
60  << component->Type();
61  component = &(nnet->GetComponent(nc - 3));
62  if ((*affine_component = dynamic_cast<AffineComponent*>(component)) == NULL)
63  KALDI_ERR << "Neural net has wrong topology: expected third-to-last "
64  << "component to be AffineComponent, type is "
65  << component->Type();
66 }
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_LOG
Definition: kaldi-error.h:133
bool kaldi::nnet2::HasSimpleLabels ( const NnetExample &  eg,
std::vector< int32 > *  simple_labels 
)

Definition at line 32 of file nnet-example.cc.

References NnetExample::labels.

Referenced by NnetExample::Write().

34  {
35  size_t num_frames = eg.labels.size();
36  for (int32 t = 0; t < num_frames; t++)
37  if (eg.labels[t].size() != 1 || eg.labels[t][0].second != 1.0)
38  return false;
39  simple_labels->resize(num_frames);
40  for (int32 t = 0; t < num_frames; t++)
41  (*simple_labels)[t] = eg.labels[t][0].first;
42  return true;
43 }
int32 IndexOfSoftmaxLayer ( const Nnet &  nnet)

If "nnet" has exactly one softmax layer, this function will return its index; otherwise it will return -1.

Definition at line 27 of file nnet-functions.cc.

References Nnet::GetComponent(), and Nnet::NumComponents().

Referenced by main().

27  {
28  int32 index = -1, nc = nnet.NumComponents();
29  for (int32 c = 0; c < nc; c++) {
30  const Component *component = &(nnet.GetComponent(c));
31  if (dynamic_cast<const SoftmaxComponent*>(component) != NULL) {
32  if (index != -1) return -1; // >1 softmax components.
33  else index = c;
34  }
35  }
36  return index;
37 }
void InsertComponents ( const Nnet &  src_nnet,
int32  c,
Nnet *  dest_nnet 
)

Inserts the components of one neural network into a particular place in the other one.

This is useful for adding hidden layers to a neural net. Inserts the components of "src_nnet" before component index c of "dest_nnet".

Definition at line 39 of file nnet-functions.cc.

References Component::Copy(), Nnet::GetComponent(), Nnet::Init(), KALDI_ASSERT, and Nnet::NumComponents().

Referenced by main().

41  {
42  KALDI_ASSERT(c_to_insert >= 0 && c_to_insert <= dest_nnet->NumComponents());
43  int32 c_tot = dest_nnet->NumComponents() + src_nnet.NumComponents();
44  std::vector<Component*> components(c_tot);
45  for (int32 c = 0; c < c_to_insert; c++)
46  components[c] = dest_nnet->GetComponent(c).Copy();
47  for (int32 c = 0; c < src_nnet.NumComponents(); c++)
48  components[c + c_to_insert] = src_nnet.GetComponent(c).Copy();
49  for (int32 c = c_to_insert; c < dest_nnet->NumComponents(); c++)
50  components[c + src_nnet.NumComponents()] = dest_nnet->GetComponent(c).Copy();
51  // Re-initialize "dest_nnet" from the resulting list of components.
52 
53  // The Init method will take ownership of the pointers in the vector:
54  dest_nnet->Init(&components);
55 }
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
BaseFloat kaldi::nnet2::KlDivergence ( const Vector< BaseFloat > &  p,
const Vector< BaseFloat > &  q 
)

Definition at line 31 of file nnet-adjust-priors.cc.

References VectorBase< Real >::Dim(), rnnlm::i, KALDI_ASSERT, KALDI_WARN, kaldi::Log(), and VectorBase< Real >::Sum().

Referenced by PrintPriorDiagnostics().

32  {
33  BaseFloat sum_p = p.Sum(), sum_q = q.Sum();
34  if (fabs(sum_p - 1.0) > 0.01 || fabs(sum_q - 1.0) > 0.01) {
35  KALDI_WARN << "KlDivergence: vectors are not close to being normalized "
36  << sum_p << ", " << sum_q;
37  }
38  KALDI_ASSERT(p.Dim() == q.Dim());
39  double ans = 0.0;
40 
41  for (int32 i = 0; i < p.Dim(); i++) {
42  BaseFloat p_prob = p(i) / sum_p, q_prob = q(i) / sum_q;
43  ans += p_prob * Log(p_prob / q_prob);
44  }
45  return ans;
46 }
Real Sum() const
Returns sum of the elements.
float BaseFloat
Definition: kaldi-types.h:29
double Log(double x)
Definition: kaldi-math.h:100
#define KALDI_WARN
Definition: kaldi-error.h:130
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
bool LatticeToDiscriminativeExample ( const std::vector< int32 > &  alignment,
const Matrix< BaseFloat > &  feats,
const CompactLattice &  clat,
BaseFloat  weight,
int32  left_context,
int32  right_context,
DiscriminativeNnetExample *  eg 
)

Converts lattice to discriminative training example.

returns true on success, false on failure such as mismatched input (will also warn in this case).

Definition at line 27 of file nnet-example-functions.cc.

References DiscriminativeNnetExample::Check(), kaldi::CompactLatticeStateTimes(), DiscriminativeNnetExample::den_lat, DiscriminativeNnetExample::input_frames, KALDI_ASSERT, KALDI_WARN, DiscriminativeNnetExample::left_context, DiscriminativeNnetExample::num_ali, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), MatrixBase< Real >::Range(), Matrix< Real >::Resize(), MatrixBase< Real >::Row(), and DiscriminativeNnetExample::weight.

Referenced by main().

34  {
35  KALDI_ASSERT(left_context >= 0 && right_context >= 0);
36  int32 num_frames = alignment.size();
37  if (num_frames == 0) {
38  KALDI_WARN << "Empty alignment";
39  return false;
40  }
41  if (num_frames != feats.NumRows()) {
42  KALDI_WARN << "Dimension mismatch: alignment " << num_frames
43  << " versus feats " << feats.NumRows();
44  return false;
45  }
46  std::vector<int32> times;
47  int32 num_frames_clat = CompactLatticeStateTimes(clat, &times);
48  if (num_frames_clat != num_frames) {
49  KALDI_WARN << "Numerator/frames versus denlat frames mismatch: "
50  << num_frames << " versus " << num_frames_clat;
51  return false;
52  }
53  eg->weight = weight;
54  eg->num_ali = alignment;
55  eg->den_lat = clat;
56 
57  int32 feat_dim = feats.NumCols();
58  eg->input_frames.Resize(left_context + num_frames + right_context,
59  feat_dim);
60  eg->input_frames.Range(left_context, num_frames,
61  0, feat_dim).CopyFromMat(feats);
62 
63  // Duplicate the first and last frames.
64  for (int32 t = 0; t < left_context; t++)
65  eg->input_frames.Row(t).CopyFromVec(feats.Row(0));
66  for (int32 t = 0; t < right_context; t++)
67  eg->input_frames.Row(left_context + num_frames + t).CopyFromVec(
68  feats.Row(num_frames - 1));
69 
70  eg->left_context = left_context;
71  eg->Check();
72  return true;
73 }
const SubVector< Real > Row(MatrixIndexT i) const
Return specific row of matrix [const].
Definition: kaldi-matrix.h:182
int32 CompactLatticeStateTimes(const CompactLattice &lat, vector< int32 > *times)
As LatticeStateTimes, but in the CompactLattice format.
#define KALDI_WARN
Definition: kaldi-error.h:130
MatrixIndexT NumRows() const
Returns number of rows (or zero for emtpy matrix).
Definition: kaldi-matrix.h:58
MatrixIndexT NumCols() const
Returns number of columns (or zero for emtpy matrix).
Definition: kaldi-matrix.h:61
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void LimitRankParallel ( const NnetLimitRankOpts &  opts,
Nnet *  nnet 
)

This function limits the rank of each affine transform in the neural net, by zeroing out the smallest singular values.

The number of singular values to zero out is determined on a layer by layer basis, using "parameter_proportion" to set the proportion of parameters to remove.

Definition at line 99 of file nnet-limit-rank.cc.

References Nnet::GetComponent(), NnetLimitRankOpts::num_threads, TaskSequencerConfig::num_threads, Nnet::NumComponents(), and TaskSequencer< C >::Run().

Referenced by main().

100  {
101  TaskSequencerConfig task_config;
102  task_config.num_threads = opts.num_threads;
103  TaskSequencer<LimitRankClass> tc(task_config);
104  for (int32 c = 0; c < nnet->NumComponents(); c++) {
105  if (dynamic_cast<AffineComponent*>(&(nnet->GetComponent(c))) != NULL)
106  tc.Run(new LimitRankClass(opts, c, nnet));
107  }
108 }
static Int32Pair kaldi::nnet2::MakePair ( int32  first,
int32  second 
)
inlinestatic

Definition at line 27 of file train-nnet-ensemble.cc.

References Int32Pair::first, and Int32Pair::second.

Referenced by NnetEnsembleTrainer::TrainOneMinibatch().

27  {
28  Int32Pair ans;
29  ans.first = first;
30  ans.second = second;
31  return ans;
32 }
int32_cuda second
Definition: cu-matrixdim.h:86
int32_cuda first
Definition: cu-matrixdim.h:85
void MixupNnet ( const NnetMixupConfig &  mixup_config,
Nnet *  nnet 
)

This function works as follows.

This function does something similar to Gaussian mixture splitting for GMMs, except applied to the output layer of the neural network.

We first make sure the neural net has the correct topology, so its last component is a SumGroupComponent.

We then get the counts for each matrix in the SumGroupComponent (these will either correspond to leaves in the decision tree, or level-1 leaves, if we have a 2-level-tree system). We work out the total count for each of these matrices, by getting the count from the SoftmaxComponent.

We then increase, if necessary, the dimensions that the SumGroupComponent sums over increase the dimension of the SoftmaxComponent if necessary, and duplicate and then perturb the relevant rows of the AffineComponent.

We create additional outputs, which will be summed over using a SumGroupComponent.

Definition at line 86 of file mixup-nnet.cc.

References Nnet::Check(), GiveNnetCorrectTopology(), NnetMixupConfig::min_count, SoftmaxComponent::MixUp(), NnetMixupConfig::num_mixtures, NnetMixupConfig::perturb_stddev, and NnetMixupConfig::power.

Referenced by main().

87  {
88  AffineComponent *affine_component = NULL;
89  SoftmaxComponent *softmax_component = NULL;
90  SumGroupComponent *sum_group_component = NULL;
92  &affine_component,
93  &softmax_component,
94  &sum_group_component); // Adds a SumGroupComponent if needed.
95 
96  softmax_component->MixUp(mixup_config.num_mixtures,
97  mixup_config.power,
98  mixup_config.min_count,
99  mixup_config.perturb_stddev,
100  affine_component,
101  sum_group_component);
102  nnet->Check(); // Checks that dimensions all match up.
103 }
static void GiveNnetCorrectTopology(Nnet *nnet, AffineComponent **affine_component, SoftmaxComponent **softmax_component, SumGroupComponent **sum_group_component)
This function makes sure the neural net ends with a SumGroupComponent.
Definition: mixup-nnet.cc:37
void NnetComputation ( const Nnet &  nnet,
const CuMatrixBase< BaseFloat > &  input,
bool  pad_input,
CuMatrixBase< BaseFloat > *  output 
)

Does the basic neural net computation, on a sequence of data (e.g.

an utterance). If pad_input==true we'll pad the input with enough frames of context, and the output will be a matrix of #frames by the output-dim of the network, typically representing state-level posteriors. If pad_input==false we won't do this and the output will have a lower #frames than the input; we lose nnet.LeftContext() at the left and nnet.RightContext() at the output.

Definition at line 160 of file nnet-compute.cc.

References CuMatrixBase< Real >::CopyFromMat(), NnetComputer::GetOutput(), and NnetComputer::Propagate().

Referenced by CachingOptimizingCompiler::CompileNoShortcut(), CachingOptimizingCompiler::CompileViaShortcut(), DecodableAmNnetParallel::Compute(), DecodableNnet2Online::ComputeForFrame(), DecodableAmNnet::DecodableAmNnet(), main(), NnetLogprobTask::operator()(), CachingOptimizingCompiler::ReadCache(), UnitTestNnetCompute(), and UnitTestNnetComputeChunked().

163  {
164  NnetComputer nnet_computer(nnet, input, pad_input, NULL);
165  nnet_computer.Propagate();
166  output->CopyFromMat(nnet_computer.GetOutput());
167 }
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
void NnetComputationChunked ( const Nnet &  nnet,
const Matrix< BaseFloat > &  input,
int32  chunk_size,
Matrix< BaseFloat > *  output 
)

Does the basic neural net computation, on a sequence of data (e.g.

an utterance). This variant of NnetComputation chunks the input according to chunk_size and does the posterior computation chunk by chunk. This allows the computation to be performed on the GPU when the input matrix is very large. Input is padded with enough frames of context so that the output will be a matrix with input.NumRows().

Definition at line 169 of file nnet-compute.cc.

References MatrixBase< Real >::CopyFromMat(), NnetComputer::GetOutput(), rnnlm::i, Nnet::LeftContext(), MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), NnetComputer::Propagate(), MatrixBase< Real >::Range(), Matrix< Real >::Resize(), Nnet::RightContext(), and MatrixBase< Real >::Row().

Referenced by main(), and UnitTestNnetComputeChunked().

172  {
173  int32 num_rows,
174  num_chunks = ceil((BaseFloat)input.NumRows() / chunk_size),
175  dim = input.NumCols(),
176  left_context = nnet.LeftContext(),
177  right_context = nnet.RightContext();
178  Matrix<BaseFloat> full_input;
179  num_rows = left_context + input.NumRows() + right_context;
180  full_input.Resize(num_rows, dim);
181  full_input.Range(left_context, input.NumRows(),
182  0, dim).CopyFromMat(input);
183  for (int32 i = 0; i < left_context; i++)
184  full_input.Row(i).CopyFromVec(input.Row(0));
185  int32 last_row = input.NumRows() - 1;
186  for (int32 i = 0; i < right_context; i++)
187  full_input.Row(num_rows - i - 1).CopyFromVec(input.Row(last_row));
188 
189  for (int32 i = 0; i < num_chunks; i++) {
190  int32 index = i * chunk_size,
191  offset = std::min(num_rows - chunk_size * i,
192  left_context + chunk_size + right_context);
193  SubMatrix<BaseFloat> chunk_input(full_input, index, offset, 0, dim);
194  CuMatrix<BaseFloat> cu_chunk_input(chunk_input);
195 
196  // Note: we have already accounted for input padding, so we pass
197  // pad_input==false to the NnetComputer.
198  NnetComputer nnet_computer(nnet, cu_chunk_input, false, NULL);
199  nnet_computer.Propagate();
200  CuMatrix<BaseFloat> cu_chunk_output(nnet_computer.GetOutput());
201  SubMatrix<BaseFloat> chunk_out(*output, i * chunk_size,
202  cu_chunk_output.NumRows(), 0,
203  cu_chunk_output.NumCols());
204  chunk_out.CopyFromMat(cu_chunk_output);
205  }
206 }
const SubVector< Real > Row(MatrixIndexT i) const
Return specific row of matrix [const].
Definition: kaldi-matrix.h:182
float BaseFloat
Definition: kaldi-types.h:29
MatrixIndexT NumRows() const
Returns number of rows (or zero for emtpy matrix).
Definition: kaldi-matrix.h:58
MatrixIndexT NumCols() const
Returns number of columns (or zero for emtpy matrix).
Definition: kaldi-matrix.h:61
void NnetDiscriminativeUpdate ( const AmNnet &  am_nnet,
const TransitionModel &  tmodel,
const NnetDiscriminativeUpdateOptions &  opts,
const DiscriminativeNnetExample &  eg,
Nnet *  nnet_to_update,
NnetDiscriminativeStats *  stats 
)

Does the neural net computation, lattice forward-backward, and backprop, for either the MMI, MPFE or SMBR objective functions.

If nnet_to_update == &(am_nnet.GetNnet()), then this does stochastic gradient descent, otherwise (assuming you have called SetZero(true) on *nnet_to_update) it will compute the gradient on this data. If nnet_to_update_ == NULL, no backpropagation is done.

Note: we ignore any existing acoustic score in the lattice of "eg".

For display purposes you should normalize the sum of this return value by dividing by the sum over the examples, of the number of frames (num_ali.size()) times the weight.

Something you need to be careful with is that the occupation counts and the derivative are, following tradition, missing a factor equal to the acoustic scale. So you need to multiply them by that scale if you plan to do something like L-BFGS in which you look at both the derivatives and function values.

Definition at line 365 of file nnet-compute-discriminative.cc.

References NnetDiscriminativeUpdater::Update().

Referenced by main(), and DiscTrainParallelClass::operator()().

370  {
371  NnetDiscriminativeUpdater updater(am_nnet, tmodel, opts, eg,
372  nnet_to_update, stats);
373  updater.Update();
374 }
void NnetDiscriminativeUpdateParallel ( const AmNnet &  am_nnet,
const TransitionModel &  tmodel,
const NnetDiscriminativeUpdateOptions &  opts,
int32  num_threads,
SequentialDiscriminativeNnetExampleReader *  example_reader,
Nnet *  nnet_to_update,
NnetDiscriminativeStats *  stats 
)

Definition at line 189 of file nnet-compute-discriminative-parallel.cc.

References DiscriminativeExamplesRepository::AcceptExample(), NnetDiscriminativeUpdateOptions::criterion, SequentialTableReader< Holder >::Done(), DiscriminativeExamplesRepository::ExamplesDone(), AmNnet::GetNnet(), SequentialTableReader< Holder >::Next(), NnetDiscriminativeStats::Print(), and SequentialTableReader< Holder >::Value().

Referenced by main().

196  {
197 
198  DiscriminativeExamplesRepository repository;
199 
200  const bool store_separate_gradients = (nnet_to_update != &(am_nnet.GetNnet()));
201 
202  DiscTrainParallelClass c(am_nnet, tmodel, opts,
203  store_separate_gradients,
204  &repository, nnet_to_update, stats);
205 
206  {
207  // The initialization of the following class spawns the threads that
208  // process the examples. They get re-joined in its destructor.
209  MultiThreader<DiscTrainParallelClass> m(num_threads, c);
210 
211  for (; !example_reader->Done(); example_reader->Next()) {
212  repository.AcceptExample(example_reader->Value());
213  }
214  repository.ExamplesDone();
215  }
216  stats->Print(opts.criterion);
217 }
BaseFloat kaldi::nnet2::NnetGradientComputation ( const Nnet &  nnet,
const MatrixBase< BaseFloat > &  input,
bool  pad_input,
BaseFloat  utterance_weight,
const std::vector< int32 > &  labels,
Nnet *  nnet_to_update 
)

Does the neural net computation and backprop, given input and labels.

Note: if pad_input==true the number of rows of input should be the same as the number of labels, and if false, you should omit nnet.LeftContext() labels on the left and nnet.RightContext() on the right. If nnet_to_update == &nnet, then this does stochastic gradient descent, otherwise (assuming you have called SetZero(true) on *nnet_to_update) it will compute the gradient on this data. Returns the total objective function summed over the frames, times the utterance weight.

BaseFloat kaldi::nnet2::NnetGradientComputation ( const Nnet &  nnet,
const CuMatrixBase< BaseFloat > &  input,
bool  pad_input,
const Posterior &  pdf_post,
Nnet *  nnet_to_update 
)

Definition at line 208 of file nnet-compute.cc.

References NnetComputer::Backprop(), NnetComputer::ComputeLastLayerDeriv(), and NnetComputer::Propagate().

212  {
213  NnetComputer nnet_computer(nnet, input, pad_input, nnet_to_update);
214  nnet_computer.Propagate();
215  CuMatrix<BaseFloat> deriv;
216  BaseFloat ans;
217  ans = nnet_computer.ComputeLastLayerDeriv(pdf_post, &deriv);
218  nnet_computer.Backprop(&deriv);
219  return ans;
220 }
float BaseFloat
Definition: kaldi-types.h:29
bool ParseFromString ( const std::string &  name,
std::string *  string,
int32 *  param 
)

Functions used in Init routines.

Suppose name=="foo", if "string" has a field like foo=12, this function will set "param" to 12 and remove that element from "string". It returns true if the parameter was read.

Definition at line 153 of file nnet-component.cc.

References kaldi::ConvertStringToInteger(), rnnlm::i, rnnlm::j, KALDI_ERR, and kaldi::SplitStringToVector().

Referenced by NonlinearComponent::InitFromString(), MaxoutComponent::InitFromString(), MaxpoolingComponent::InitFromString(), PnormComponent::InitFromString(), PowerComponent::InitFromString(), ScaleComponent::InitFromString(), AffineComponent::InitFromString(), AffineComponentPreconditioned::InitFromString(), AffineComponentPreconditionedOnline::InitFromString(), SpliceComponent::InitFromString(), SpliceMaxComponent::InitFromString(), BlockAffineComponent::InitFromString(), BlockAffineComponentPreconditioned::InitFromString(), SumGroupComponent::InitFromString(), PermuteComponent::InitFromString(), DctComponent::InitFromString(), FixedLinearComponent::InitFromString(), FixedAffineComponent::InitFromString(), FixedScaleComponent::InitFromString(), FixedBiasComponent::InitFromString(), DropoutComponent::InitFromString(), AdditiveNoiseComponent::InitFromString(), Convolutional1dComponent::InitFromString(), and UnitTestParsing().

154  {
155  std::vector<std::string> split_string;
156  SplitStringToVector(*string, " \t", true,
157  &split_string);
158  std::string name_equals = name + "="; // the name and then the equals sign.
159  size_t len = name_equals.length();
160 
161  for (size_t i = 0; i < split_string.size(); i++) {
162  if (split_string[i].compare(0, len, name_equals) == 0) {
163  if (!ConvertStringToInteger(split_string[i].substr(len), param))
164  KALDI_ERR << "Bad option " << split_string[i];
165  *string = "";
166  // Set "string" to all the pieces but the one we used.
167  for (size_t j = 0; j < split_string.size(); j++) {
168  if (j != i) {
169  if (!string->empty()) *string += " ";
170  *string += split_string[j];
171  }
172  }
173  return true;
174  }
175  }
176  return false;
177 }
bool ConvertStringToInteger(const std::string &str, Int *out)
Converts a string into an integer via strtoll and returns false if there was any kind of problem (i...
Definition: text-utils.h:118
void SplitStringToVector(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< std::string > *out)
Split a string using any of the single character delimiters.
Definition: text-utils.cc:63
#define KALDI_ERR
Definition: kaldi-error.h:127
bool ParseFromString ( const std::string &  name,
std::string *  string,
bool *  param 
)

This version is for parameters of type bool, which can appear as any string beginning with f, F, t or T.

Definition at line 179 of file nnet-component.cc.

References rnnlm::i, rnnlm::j, KALDI_ERR, and kaldi::SplitStringToVector().

180  {
181  std::vector<std::string> split_string;
182  SplitStringToVector(*string, " \t", true,
183  &split_string);
184  std::string name_equals = name + "="; // the name and then the equals sign.
185  size_t len = name_equals.length();
186 
187  for (size_t i = 0; i < split_string.size(); i++) {
188  if (split_string[i].compare(0, len, name_equals) == 0) {
189  std::string b = split_string[i].substr(len);
190  if (b.empty())
191  KALDI_ERR << "Bad option " << split_string[i];
192  if (b[0] == 'f' || b[0] == 'F') *param = false;
193  else if (b[0] == 't' || b[0] == 'T') *param = true;
194  else
195  KALDI_ERR << "Bad option " << split_string[i];
196  *string = "";
197  // Set "string" to all the pieces but the one we used.
198  for (size_t j = 0; j < split_string.size(); j++) {
199  if (j != i) {
200  if (!string->empty()) *string += " ";
201  *string += split_string[j];
202  }
203  }
204  return true;
205  }
206  }
207  return false;
208 }
void SplitStringToVector(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< std::string > *out)
Split a string using any of the single character delimiters.
Definition: text-utils.cc:63
#define KALDI_ERR
Definition: kaldi-error.h:127
bool ParseFromString ( const std::string &  name,
std::string *  string,
BaseFloat *  param 
)

This version is for parameters of type BaseFloat.

Definition at line 210 of file nnet-component.cc.

References kaldi::ConvertStringToReal(), rnnlm::i, rnnlm::j, KALDI_ERR, and kaldi::SplitStringToVector().

211  {
212  std::vector<std::string> split_string;
213  SplitStringToVector(*string, " \t", true,
214  &split_string);
215  std::string name_equals = name + "="; // the name and then the equals sign.
216  size_t len = name_equals.length();
217 
218  for (size_t i = 0; i < split_string.size(); i++) {
219  if (split_string[i].compare(0, len, name_equals) == 0) {
220  if (!ConvertStringToReal(split_string[i].substr(len), param))
221  KALDI_ERR << "Bad option " << split_string[i];
222  *string = "";
223  // Set "string" to all the pieces but the one we used.
224  for (size_t j = 0; j < split_string.size(); j++) {
225  if (j != i) {
226  if (!string->empty()) *string += " ";
227  *string += split_string[j];
228  }
229  }
230  return true;
231  }
232  }
233  return false;
234 }
void SplitStringToVector(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< std::string > *out)
Split a string using any of the single character delimiters.
Definition: text-utils.cc:63
#define KALDI_ERR
Definition: kaldi-error.h:127
bool ConvertStringToReal(const std::string &str, T *out)
ConvertStringToReal converts a string into either float or double and returns false if there was any ...
Definition: text-utils.cc:238
bool kaldi::nnet2::ParseFromString ( const std::string &  name,
std::string *  string,
std::string *  param 
)

Definition at line 236 of file nnet-component.cc.

References rnnlm::i, rnnlm::j, and kaldi::SplitStringToVector().

237  {
238  std::vector<std::string> split_string;
239  SplitStringToVector(*string, " \t", true,
240  &split_string);
241  std::string name_equals = name + "="; // the name and then the equals sign.
242  size_t len = name_equals.length();
243 
244  for (size_t i = 0; i < split_string.size(); i++) {
245  if (split_string[i].compare(0, len, name_equals) == 0) {
246  *param = split_string[i].substr(len);
247 
248  // Set "string" to all the pieces but the one we used.
249  *string = "";
250  for (size_t j = 0; j < split_string.size(); j++) {
251  if (j != i) {
252  if (!string->empty()) *string += " ";
253  *string += split_string[j];
254  }
255  }
256  return true;
257  }
258  }
259  return false;
260 }
void SplitStringToVector(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< std::string > *out)
Split a string using any of the single character delimiters.
Definition: text-utils.cc:63
bool ParseFromString ( const std::string &  name,
std::string *  string,
std::vector< int32 > *  param 
)

This version is for parameters of type std::vector<int32>; it expects them as a colon-separated list, without spaces.

Definition at line 262 of file nnet-component.cc.

References rnnlm::i, rnnlm::j, KALDI_ERR, kaldi::SplitStringToIntegers(), and kaldi::SplitStringToVector().

263  {
264  std::vector<std::string> split_string;
265  SplitStringToVector(*string, " \t", true,
266  &split_string);
267  std::string name_equals = name + "="; // the name and then the equals sign.
268  size_t len = name_equals.length();
269 
270  for (size_t i = 0; i < split_string.size(); i++) {
271  if (split_string[i].compare(0, len, name_equals) == 0) {
272  if (!SplitStringToIntegers(split_string[i].substr(len), ":",
273  false, param))
274  KALDI_ERR << "Bad option " << split_string[i];
275  *string = "";
276  // Set "string" to all the pieces but the one we used.
277  for (size_t j = 0; j < split_string.size(); j++) {
278  if (j != i) {
279  if (!string->empty()) *string += " ";
280  *string += split_string[j];
281  }
282  }
283  return true;
284  }
285  }
286  return false;
287 }
bool SplitStringToIntegers(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< I > *out)
Split a string (e.g.
Definition: text-utils.h:68
void SplitStringToVector(const std::string &full, const char *delim, bool omit_empty_strings, std::vector< std::string > *out)
Split a string using any of the single character delimiters.
Definition: text-utils.cc:63
#define KALDI_ERR
Definition: kaldi-error.h:127
void kaldi::nnet2::PerturbTrainingExample ( const TpMatrix< BaseFloat > &  cholesky,
BaseFloat  noise_factor,
NnetExample eg 
)

Definition at line 28 of file nnet-perturb-egs.cc.

References VectorBase< Real >::AddTpVec(), CompressedMatrix::CopyFromMat(), VectorBase< Real >::CopyRowsFromMat(), MatrixBase< Real >::CopyRowsFromVec(), NnetExample::input_frames, KALDI_ERR, kaldi::kNoTrans, kaldi::kUndefined, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), PackedMatrix< Real >::NumRows(), and VectorBase< Real >::SetRandn().

Referenced by main().

30  {
31  Matrix<BaseFloat> input_frames(eg->input_frames);
32  int32 dim = input_frames.NumRows() * input_frames.NumCols();
33  if (dim != cholesky.NumRows()) {
34  KALDI_ERR << "Dimension mismatch: egs have total dim " << dim
35  << " vs. cholesky factor " << cholesky.NumRows();
36  }
37  Vector<BaseFloat> vec(dim, kUndefined);
38  vec.CopyRowsFromMat(input_frames);
39  Vector<BaseFloat> noise(dim);
40  noise.SetRandn(); // Gaussian noise with unit variance and zero mean
41  vec.AddTpVec(noise_factor, cholesky, kNoTrans, noise, 1.0);
42  input_frames.CopyRowsFromVec(vec);
43  eg->input_frames.CopyFromMat(input_frames);
44 }
CompressedMatrix input_frames
The input data, with NumRows() >= labels.size() + left_context; it includes features to the left and ...
Definition: nnet-example.h:49
MatrixIndexT NumRows() const
#define KALDI_ERR
Definition: kaldi-error.h:127
void CopyFromMat(const MatrixBase< Real > &mat, CompressionMethod method=kAutomaticMethod)
This will resize *this and copy the contents of mat to *this.
void PreconditionDirections ( const CuMatrixBase< BaseFloat > &  R,
double  lambda,
CuMatrixBase< BaseFloat > *  P 
)

See below for comment.

The function PreconditionDirections views the input R as a set of directions or gradients, each row r_i being one of the directions.

For each i it constructs a preconditioning matrix G_i formed from the *other* i's, using the formula:

G_i = ( I + (1/(N-1)) {j i} r_j r_j^T)^{-1},

where N is the number of rows in R. This can be seen as a kind of estimated Fisher matrix that has been smoothed with the identity to make it invertible. We recommend that you set using: = /(N D) trace(R^T, R) for some small such as = 0.1. However, we leave this to the caller because there are reasons relating to unbiased-ness of the resulting stochastic gradient descent, why you might want to set using "other" data, e.g. a previous minibatch.

The output of this function is a matrix P, each row p_i of which is related to r_i by: p_i = G_i r_i Here, p_i is preconditioned by an estimated Fisher matrix in such a way that it's suitable to be used as an update direction.

Definition at line 26 of file nnet-precondition.cc.

References CuVectorBase< Real >::AddDiagMatMat(), CuMatrixBase< Real >::AddMatMat(), CuMatrixBase< Real >::AddToDiag(), CuMatrixBase< Real >::CopyFromMat(), CuMatrixBase< Real >::CopyLowerToUpper(), kaldi::GetVerboseLevel(), KALDI_ASSERT, KALDI_ERR, KALDI_WARN, kaldi::kNoTrans, kaldi::kTakeLower, kaldi::kTrans, kaldi::kUndefined, CuMatrixBase< Real >::MulRowsVec(), rnnlm::n, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), SpMatrix< Real >::PrintEigs(), kaldi::Rand(), kaldi::SameDim(), CuVectorBase< Real >::Scale(), CuMatrixBase< Real >::SymAddMat2(), CuMatrixBase< Real >::SymInvertPosDef(), and kaldi::VecVec().

Referenced by PreconditionDirectionsAlpha(), PreconditionDirectionsAlphaRescaled(), and UnitTestPreconditionDirections().

28  {
29 
30  int32 N = R.NumRows(), D = R.NumCols();
31  KALDI_ASSERT(SameDim(R, *P) && N > 0);
32  if (N == 1) {
33  KALDI_WARN << "Trying to precondition set of only one frames: returning "
34  << "unchanged. Ignore this warning if infrequent.";
35  P->CopyFromMat(R);
36  return;
37  }
38  CuMatrixBase<BaseFloat> &Q = *P;
39 
40  if (N >= D) {
41  // Compute G = (\lambda I + 1/(N-1) R^T R)^{-1} by direct inversion.
42  // G <-- lambda I.
43  CuMatrix<BaseFloat> G(D, D);
44  G.AddToDiag(lambda);
45  // G += 1.0/(N-1) * R^T R.
46  G.SymAddMat2(1.0 / (N-1), R, kTrans, 1.0);
47  G.CopyLowerToUpper();
48  if (GetVerboseLevel() >= 5 && Rand() % 20 == 0) {
49  CuSpMatrix<BaseFloat> tmp(G, kTakeLower);
50  SpMatrix<BaseFloat> G_cpu(tmp);
51  G_cpu.PrintEigs("G");
52  }
53  G.SymInvertPosDef();
54  // Q <-- R G^T (we just make it transposed as we think
55  // it will be slightly faster; it's symmetric).
56  Q.AddMatMat(1.0, R, kNoTrans, G, kTrans, 0.0);
57  } else {
58  // Through a lot of rearrangements, it turns out that
59  // if we let S = (\lambda I + 1/(N-1) R R^T)^{-1}
60  // then what we need is
61  // Q <-- S R.
62  // It is curious and (to me) unexpected that the actual code is basically
63  // the same when transposed.
64  CuMatrix<BaseFloat> S(N, N);
65  // S <-- lambda I.
66  S.AddToDiag(lambda);
67  // S += (N-1) R R^T.
68  // the following function only updates the lower triangle.
69  S.SymAddMat2(1.0 / (N-1), R, kNoTrans, 1.0);
70  S.CopyLowerToUpper();
71  // invert S, so now S = (\lambda I + (N-1) R R^T)^{-1}.
72  if (GetVerboseLevel() >= 5 && Rand() % 20 == 0) {
73  CuSpMatrix<BaseFloat> tmp(S, kTakeLower);
74  SpMatrix<BaseFloat> S_cpu(tmp);
75  S_cpu.PrintEigs("S");
76  }
77  S.SymInvertPosDef();
78  Q.AddMatMat(1.0, S, kNoTrans, R, kNoTrans, 0.0);
79  }
80 
81 #if 0 // Old code before it was optimized for CUDA:
82  for (int32 n = 0; n < N; n++) {
83  CuSubVector<BaseFloat> r(R, n), q(Q, n);
84  BaseFloat gamma = VecVec(r, q), // gamma_n = r_n^T q_n.
85  beta = 1 + gamma / (N - 1 - gamma);
86  if (!(gamma >= 0.0 && beta > 0.0)) {
87  KALDI_ERR << "Bad values encountered in preconditioning: gamma = " << gamma
88  << ", beta = " << beta;
89  }
90  // Q and P share the same memory. The result of the
91  // scaling below will be output as P.
92  q.Scale(beta);
93  }
94 #else
95  CuVector<BaseFloat> gamma(N);
96  gamma.AddDiagMatMat(1.0, R, kNoTrans, Q, kTrans, 0.0);
97  // at this point, gamma(i) equals the i'th row of R dotted with
98  // the i'th row of Q.
99  Vector<BaseFloat> cpu_gamma(gamma), cpu_beta(N, kUndefined);
100  for (int32 n = 0; n < N; n++) {
101  BaseFloat this_gamma = cpu_gamma(n),
102  this_beta = 1.0 + this_gamma / (N - 1 - this_gamma);
103  if (!(this_gamma >= 0.0 && this_beta > 0.0))
104  KALDI_ERR << "Bad values encountered in preconditioning: gamma = "
105  << this_gamma << ", beta = " << this_beta;
106  cpu_beta(n) = this_beta;
107  }
108  CuVector<BaseFloat> beta(cpu_beta);
109  P->MulRowsVec(beta);
110 #endif
111 }
int32 GetVerboseLevel()
Definition: kaldi-error.h:69
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
void MulRowsVec(const CuVectorBase< Real > &scale)
scale i'th row by scale[i]
Definition: cu-matrix.cc:777
bool SameDim(const MatrixBase< Real > &M, const MatrixBase< Real > &N)
float BaseFloat
Definition: kaldi-types.h:29
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
struct rnnlm::@11::@12 n
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_WARN
Definition: kaldi-error.h:130
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
Real VecVec(const VectorBase< Real > &a, const VectorBase< Real > &b)
Returns dot product between v1 and v2.
Definition: kaldi-vector.cc:36
void PreconditionDirectionsAlpha ( const CuMatrixBase< BaseFloat > &  R,
double  alpha,
CuMatrixBase< BaseFloat > *  P 
)

This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections.

Definition at line 114 of file nnet-precondition.cc.

References KALDI_ASSERT, KALDI_WARN, kaldi::kTrans, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), PreconditionDirections(), and kaldi::TraceMatMat().

117  {
118  KALDI_ASSERT(alpha > 0.0);
119  // probably does not really make sense.
120  double t = TraceMatMat(R, R, kTrans), floor = 1.0e-20;
121  if (t < floor) {
122  KALDI_WARN << "Flooring trace from " << t
123  << " to " << floor;
124  t = floor;
125  }
126  double lambda = t * alpha / R.NumRows() / R.NumCols();
127  // see the extended comment below for an explanation of this.
128  if (lambda <= 0.0) {
129  // This should never really happen, it would probably indicate a bug
130  // in the calling code.
131  KALDI_WARN << "Zero or negative lambda in PreconditionDirectionsAlpha.";
132  lambda = 1.0e-10;
133  }
134  PreconditionDirections(R, lambda, P);
135 }
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void PreconditionDirections(const CuMatrixBase< BaseFloat > &R, double lambda, CuMatrixBase< BaseFloat > *P)
See below for comment.
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_WARN
Definition: kaldi-error.h:130
Real TraceMatMat(const MatrixBase< Real > &A, const MatrixBase< Real > &B, MatrixTransposeType trans)
We need to declare this here as it will be a friend function.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void PreconditionDirectionsAlphaRescaled ( const CuMatrixBase< BaseFloat > &  R,
double  alpha,
CuMatrixBase< BaseFloat > *  P 
)

This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections.

It then rescales *P so that its 2-norm is the same as that of R.

Definition at line 138 of file nnet-precondition.cc.

References CuMatrixBase< Real >::CopyFromMat(), KALDI_ASSERT, KALDI_WARN, kaldi::kTrans, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), PreconditionDirections(), CuMatrixBase< Real >::Scale(), and kaldi::TraceMatMat().

Referenced by AffineComponentPreconditioned::Update(), and BlockAffineComponentPreconditioned::Update().

141  {
142  KALDI_ASSERT(alpha > 0.0); // alpha > 1.0
143  // probably does not really make sense.
144  double t = TraceMatMat(R, R, kTrans), floor = 1.0e-20;
145  if (t == 0.0) {
146  P->CopyFromMat(R);
147  return;
148  }
149  if (t < floor) {
150  KALDI_WARN << "Flooring trace from " << t
151  << " to " << floor;
152  t = floor;
153  }
154  double lambda = t * alpha / R.NumRows() / R.NumCols();
155  // see the extended comment below for an explanation of this.
156  KALDI_ASSERT(lambda != 0.0);
157  PreconditionDirections(R, lambda, P);
158  double p_trace = TraceMatMat(*P, *P, kTrans),
159  rescale = sqrt(t / p_trace);
160  KALDI_ASSERT(p_trace != 0.0);
161  P->Scale(rescale);
162 }
void Scale(Real value)
Definition: cu-matrix.cc:608
MatrixIndexT NumCols() const
Definition: cu-matrix.h:196
void PreconditionDirections(const CuMatrixBase< BaseFloat > &R, double lambda, CuMatrixBase< BaseFloat > *P)
See below for comment.
void CopyFromMat(const MatrixBase< OtherReal > &src, MatrixTransposeType trans=kNoTrans)
Definition: cu-matrix.cc:337
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
#define KALDI_WARN
Definition: kaldi-error.h:130
Real TraceMatMat(const MatrixBase< Real > &A, const MatrixBase< Real > &B, MatrixTransposeType trans)
We need to declare this here as it will be a friend function.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void kaldi::nnet2::PrintPriorDiagnostics ( const Vector< BaseFloat > &  old_priors,
const Vector< BaseFloat > &  new_priors 
)

Definition at line 48 of file nnet-adjust-priors.cc.

References VectorBase< Real >::AddVec(), VectorBase< Real >::ApplyAbs(), VectorBase< Real >::Dim(), KALDI_LOG, KlDivergence(), and VectorBase< Real >::Max().

Referenced by main().

49  {
50  if (old_priors.Dim() == 0) {
51  KALDI_LOG << "Model did not previously have priors attached.";
52  } else {
53  Vector<BaseFloat> diff_prior(new_priors);
54  diff_prior.AddVec(-1.0, old_priors);
55  diff_prior.ApplyAbs();
56  int32 max_index;
57  diff_prior.Max(&max_index);
58  KALDI_LOG << "Adjusting priors: largest absolute difference was for "
59  << "pdf " << max_index << ", " << old_priors(max_index)
60  << " -> " << new_priors(max_index);
61  KALDI_LOG << "Adjusting priors: K-L divergence from old to new is "
62  << KlDivergence(old_priors, new_priors);
63  }
64 }
BaseFloat KlDivergence(const Vector< BaseFloat > &p, const Vector< BaseFloat > &q)
#define KALDI_LOG
Definition: kaldi-error.h:133
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
static void kaldi::nnet2::ProcessFile ( const MatrixBase< BaseFloat > &  feats,
const Posterior pdf_post,
const std::string &  utt_id,
int32  left_context,
int32  right_context,
int32  num_frames,
int32  const_feat_dim,
int64 *  num_frames_written,
int64 *  num_egs_written,
NnetExampleWriter example_writer 
)
static

Definition at line 32 of file nnet-get-egs.cc.

References VectorBase< Real >::AddVec(), NnetExample::input_frames, rnnlm::j, KALDI_ASSERT, NnetExample::labels, NnetExample::left_context, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), Vector< Real >::Resize(), MatrixBase< Real >::Row(), NnetExample::spk_info, and TableWriter< Holder >::Write().

Referenced by main().

41  {
42  KALDI_ASSERT(feats.NumRows() == static_cast<int32>(pdf_post.size()));
43  int32 feat_dim = feats.NumCols();
44  KALDI_ASSERT(const_feat_dim < feat_dim);
45  KALDI_ASSERT(num_frames > 0);
46  int32 basic_feat_dim = feat_dim - const_feat_dim;
47 
48  for (int32 t = 0; t < feats.NumRows(); t += num_frames) {
49  int32 this_num_frames = std::min(num_frames,
50  feats.NumRows() - t);
51 
52  int32 tot_frames = left_context + this_num_frames + right_context;
53  NnetExample eg;
54  Matrix<BaseFloat> input_frames(tot_frames, basic_feat_dim);
55  eg.left_context = left_context;
56  eg.spk_info.Resize(const_feat_dim);
57 
58  // Set up "input_frames".
59  for (int32 j = -left_context; j < this_num_frames + right_context; j++) {
60  int32 t2 = j + t;
61  if (t2 < 0) t2 = 0;
62  if (t2 >= feats.NumRows()) t2 = feats.NumRows() - 1;
63  SubVector<BaseFloat> src(feats.Row(t2), 0, basic_feat_dim),
64  dest(input_frames, j + left_context);
65  dest.CopyFromVec(src);
66  if (const_feat_dim > 0) {
67  SubVector<BaseFloat> src(feats.Row(t2), basic_feat_dim, const_feat_dim);
68  // set eg.spk_info to the average of the corresponding dimensions of
69  // the input, taken over the frames whose features we store in the eg.
70  eg.spk_info.AddVec(1.0 / tot_frames, src);
71  }
72  }
73  eg.labels.resize(this_num_frames);
74  for (int32 j = 0; j < this_num_frames; j++)
75  eg.labels[j] = pdf_post[t + j];
76  eg.input_frames = input_frames; // Copy to CompressedMatrix.
77 
78  std::ostringstream os;
79  os << utt_id << "-" << t;
80 
81  std::string key = os.str(); // key is <utt_id>-<frame_id>
82 
83  *num_frames_written += this_num_frames;
84  *num_egs_written += 1;
85 
86  example_writer->Write(key, eg);
87  }
88 }
CompressedMatrix input_frames
The input data, with NumRows() >= labels.size() + left_context; it includes features to the left and ...
Definition: nnet-example.h:49
NnetExample is the input data and corresponding label (or labels) for one or more frames of input...
Definition: nnet-example.h:36
void Write(const std::string &key, const T &value) const
int32 left_context
The number of frames of left context (we can work out the #frames of right context from input_frames...
Definition: nnet-example.h:53
void Resize(MatrixIndexT length, MatrixResizeType resize_type=kSetZero)
Set vector to a specified size (can be zero).
const SubVector< Real > Row(MatrixIndexT i) const
Return specific row of matrix [const].
Definition: kaldi-matrix.h:182
MatrixIndexT NumRows() const
Returns number of rows (or zero for emtpy matrix).
Definition: kaldi-matrix.h:58
MatrixIndexT NumCols() const
Returns number of columns (or zero for emtpy matrix).
Definition: kaldi-matrix.h:61
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
std::vector< std::vector< std::pair< int32, BaseFloat > > > labels
The label(s) for each frame in a sequence of frames; in the normal case, this will be just [ [ (pdf-i...
Definition: nnet-example.h:43
void AddVec(const Real alpha, const VectorBase< OtherReal > &v)
Add vector : *this = *this + alpha * rv (with casting between floats and doubles) ...
Represents a non-allocating general vector which can be defined as a sub-vector of higher-level vecto...
Definition: kaldi-vector.h:482
Vector< BaseFloat > spk_info
The speaker-specific input, if any, or an empty vector if we're not using this features.
Definition: nnet-example.h:58
static void kaldi::nnet2::ProcessFile ( const MatrixBase< BaseFloat > &  feats,
const Posterior pdf_post,
const std::string &  utt_id,
const Vector< BaseFloat > &  weights,
int32  left_context,
int32  right_context,
int32  const_feat_dim,
BaseFloat  keep_proportion,
BaseFloat  weight_threshold,
bool  use_frame_selection,
bool  use_frame_weights,
int64 *  num_frames_written,
int64 *  num_frames_skipped,
NnetExampleWriter example_writer 
)
static

Definition at line 45 of file nnet-get-weighted-egs.cc.

References VectorBase< Real >::CopyFromVec(), count, GetCount(), rnnlm::i, NnetExample::input_frames, rnnlm::j, KALDI_ASSERT, NnetExample::labels, NnetExample::left_context, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), MatrixBase< Real >::Row(), NnetExample::spk_info, and TableWriter< Holder >::Write().

58  {
59  KALDI_ASSERT(feats.NumRows() == static_cast<int32>(pdf_post.size()));
60  int32 feat_dim = feats.NumCols();
61  KALDI_ASSERT(const_feat_dim < feat_dim);
62  int32 basic_feat_dim = feat_dim - const_feat_dim;
63  NnetExample eg;
64  Matrix<BaseFloat> input_frames(left_context + 1 + right_context,
65  basic_feat_dim);
66  eg.left_context = left_context;
67  // TODO: modify this code, and this binary itself, to support the --num-frames
68  // option to allow multiple frames per eg.
69  for (int32 i = 0; i < feats.NumRows(); i++) {
70  int32 count = GetCount(keep_proportion); // number of times
71  // we'll write this out (1 by default).
72  if (count > 0) {
73  // Set up "input_frames".
74  for (int32 j = -left_context; j <= right_context; j++) {
75  int32 j2 = j + i;
76  if (j2 < 0) j2 = 0;
77  if (j2 >= feats.NumRows()) j2 = feats.NumRows() - 1;
78  SubVector<BaseFloat> src(feats, j2), dest(input_frames,
79  j + left_context);
80  dest.CopyFromVec(src);
81  }
82  eg.labels.push_back(pdf_post[i]);
83  eg.input_frames = input_frames;
84  if (const_feat_dim > 0) {
85  // we'll normally reach here if we're using online-estimated iVectors.
86  SubVector<BaseFloat> const_part(feats.Row(i),
87  basic_feat_dim, const_feat_dim);
88  eg.spk_info.CopyFromVec(const_part);
89  }
90  if (use_frame_selection) {
91  if (weights(i) < weight_threshold) {
92  (*num_frames_skipped)++;
93  continue;
94  }
95  }
96  std::ostringstream os;
97  os << utt_id << "-" << i;
98  std::string key = os.str(); // key in the archive is the number of the example
99 
100  for (int32 c = 0; c < count; c++)
101  example_writer->Write(key, eg);
102  }
103  }
104 }
CompressedMatrix input_frames
The input data, with NumRows() >= labels.size() + left_context; it includes features to the left and ...
Definition: nnet-example.h:49
NnetExample is the input data and corresponding label (or labels) for one or more frames of input...
Definition: nnet-example.h:36
void Write(const std::string &key, const T &value) const
int32 left_context
The number of frames of left context (we can work out the #frames of right context from input_frames...
Definition: nnet-example.h:53
const SubVector< Real > Row(MatrixIndexT i) const
Return specific row of matrix [const].
Definition: kaldi-matrix.h:182
void CopyFromVec(const VectorBase< Real > &v)
Copy data from another vector (must match own size).
const size_t count
MatrixIndexT NumRows() const
Returns number of rows (or zero for emtpy matrix).
Definition: kaldi-matrix.h:58
MatrixIndexT NumCols() const
Returns number of columns (or zero for emtpy matrix).
Definition: kaldi-matrix.h:61
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
std::vector< std::vector< std::pair< int32, BaseFloat > > > labels
The label(s) for each frame in a sequence of frames; in the normal case, this will be just [ [ (pdf-i...
Definition: nnet-example.h:43
int32 GetCount(double expected_count)
Represents a non-allocating general vector which can be defined as a sub-vector of higher-level vecto...
Definition: kaldi-vector.h:482
Vector< BaseFloat > spk_info
The speaker-specific input, if any, or an empty vector if we're not using this features.
Definition: nnet-example.h:58
void ReplaceLastComponents ( const Nnet &  src_nnet,
int32  num_to_remove,
Nnet *  dest_nnet 
)

Removes the last "num_to_remove" components and adds the components from "src_nnet".

Definition at line 58 of file nnet-functions.cc.

References Component::Copy(), Nnet::GetComponent(), Nnet::Init(), KALDI_ASSERT, and Nnet::NumComponents().

Referenced by main().

60  {
61  KALDI_ASSERT(num_to_remove >= 0 && num_to_remove <= dest_nnet->NumComponents());
62  int32 c_orig = dest_nnet->NumComponents() - num_to_remove;
63 
64  std::vector<Component*> components;
65  for (int32 c = 0; c < c_orig; c++)
66  components.push_back(dest_nnet->GetComponent(c).Copy());
67  for (int32 c = 0; c < src_nnet.NumComponents(); c++)
68  components.push_back(src_nnet.GetComponent(c).Copy());
69 
70  // Re-initialize "dest_nnet" from the resulting list of components.
71  // The Init method will take ownership of the pointers in the vector:
72  dest_nnet->Init(&components);
73 }
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void RescaleNnet ( const NnetRescaleConfig &  rescale_config,
const std::vector< NnetExample > &  examples,
Nnet *  nnet 
)

Definition at line 218 of file rescale-nnet.cc.

References NnetRescaler::Rescale().

Referenced by main().

220  {
221  NnetRescaler rescaler(rescale_config, examples, nnet);
222  rescaler.Rescale();
223 }
void kaldi::nnet2::SetMaxChange ( BaseFloat  max_change,
Nnet nnet 
)

Definition at line 29 of file nnet-modify-learning-rates.cc.

References Nnet::GetComponent(), Nnet::NumComponents(), and AffineComponentPreconditioned::SetMaxChange().

Referenced by main().

29  {
30  for (int32 c = 0; c < nnet->NumComponents(); c++) {
31  Component *component = &(nnet->GetComponent(c));
33  dynamic_cast<AffineComponentPreconditioned*>(component);
34  if (ac != NULL)
35  ac->SetMaxChange(max_change);
36  }
37 }
Abstract class, basic element of the network, it is a box with defined inputs, outputs, and tranformation functions interface.
int32 NumComponents() const
Returns number of components– think of this as similar to # of layers, but e.g.
Definition: nnet-nnet.h:69
const Component & GetComponent(int32 c) const
Definition: nnet-nnet.cc:141
void kaldi::nnet2::SetPriors ( const TransitionModel tmodel,
const Vector< double > &  transition_accs,
double  prior_floor,
AmNnet am_nnet 
)

Definition at line 28 of file nnet-train-transitions.cc.

References VectorBase< Real >::Dim(), KALDI_ASSERT, AmNnet::NumPdfs(), TransitionModel::NumPdfs(), AmNnet::SetPriors(), and TransitionModel::TransitionIdToPdf().

Referenced by main().

31  {
32  KALDI_ASSERT(tmodel.NumPdfs() == am_nnet->NumPdfs());
33  Vector<BaseFloat> pdf_counts(tmodel.NumPdfs());
34  KALDI_ASSERT(transition_accs(0) == 0.0); // There is
35  // no zero transition-id.
36  for (int32 tid = 1; tid < transition_accs.Dim(); tid++) {
37  int32 pdf = tmodel.TransitionIdToPdf(tid);
38  pdf_counts(pdf) += transition_accs(tid);
39  }
40  BaseFloat sum = pdf_counts.Sum();
41  KALDI_ASSERT(sum != 0.0);
42  KALDI_ASSERT(prior_floor > 0.0 && prior_floor < 1.0);
43  pdf_counts.Scale(1.0 / sum);
44  pdf_counts.ApplyFloor(prior_floor);
45  pdf_counts.Scale(1.0 / pdf_counts.Sum()); // normalize again.
46  am_nnet->SetPriors(pdf_counts);
47 }
int32 TransitionIdToPdf(int32 trans_id) const
float BaseFloat
Definition: kaldi-types.h:29
int32 NumPdfs() const
Definition: am-nnet.h:55
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SetPriors(const VectorBase< BaseFloat > &priors)
Definition: am-nnet.cc:44
MatrixIndexT Dim() const
Returns the dimension of the vector.
Definition: kaldi-vector.h:62
void ShrinkNnet ( const NnetShrinkConfig &  shrink_config,
const std::vector< NnetExample > &  validation_set,
Nnet *  nnet 
)

Definition at line 66 of file shrink-nnet.cc.

References VectorBase< Real >::ApplyExp(), ComputeObjfAndGradient(), OptimizeLbfgs< Real >::DoStep(), LbfgsOptions::first_step_length, OptimizeLbfgs< Real >::GetProposedValue(), OptimizeLbfgs< Real >::GetValue(), rnnlm::i, NnetShrinkConfig::initial_step, KALDI_ASSERT, KALDI_LOG, KALDI_VLOG, LbfgsOptions::m, LbfgsOptions::minimize, NnetShrinkConfig::num_bfgs_iters, Nnet::NumUpdatableComponents(), and Nnet::ScaleComponents().

Referenced by main().

68  {
69 
70  int32 dim = nnet->NumUpdatableComponents();
71  KALDI_ASSERT(dim > 0);
72  Vector<double> log_scale(dim), gradient(dim); // will be zero.
73 
74  // Get initial gradient.
75  double objf, initial_objf;
76 
77 
78  LbfgsOptions lbfgs_options;
79  lbfgs_options.minimize = false; // We're maximizing.
80  lbfgs_options.m = dim; // Store the same number of vectors as the dimension
81  // itself, so this is BFGS.
82  lbfgs_options.first_step_length = shrink_config.initial_step;
83 
84  OptimizeLbfgs<double> lbfgs(log_scale,
85  lbfgs_options);
86 
87  for (int32 i = 0; i < shrink_config.num_bfgs_iters; i++) {
88  log_scale.CopyFromVec(lbfgs.GetProposedValue());
89  objf = ComputeObjfAndGradient(validation_set, log_scale,
90  *nnet,
91  &gradient);
92 
93  KALDI_VLOG(2) << "log-scale = " << log_scale << ", objf = " << objf
94  << ", gradient = " << gradient;
95  if (i == 0) initial_objf = objf;
96 
97  lbfgs.DoStep(objf, gradient);
98  }
99 
100  log_scale.CopyFromVec(lbfgs.GetValue(&objf));
101 
102  Vector<BaseFloat> scale(log_scale);
103  scale.ApplyExp();
104  KALDI_LOG << "Shrinking nnet, validation objf per frame changed from "
105  << initial_objf << " to " << objf << ", scale factors per layer are "
106  << scale;
107  nnet->ScaleComponents(scale);
108 }
static BaseFloat ComputeObjfAndGradient(const std::vector< NnetExample > &validation_set, const Vector< double > &log_scale_params, const Nnet &nnet, Vector< double > *gradient)
Definition: shrink-nnet.cc:25
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_VLOG(v)
Definition: kaldi-error.h:136
#define KALDI_LOG
Definition: kaldi-error.h:133
void SolvePackingProblem ( BaseFloat  max_cost,
const std::vector< BaseFloat > &  costs,
std::vector< std::vector< size_t > > *  groups 
)

This function solves the "packing problem" using the "first fit" algorithm.

It groups together the indices 0 through sizes.size() - 1, such that the sum of cost within each group does not exceed max_lcost. [However, if there are single examples that exceed max_cost, it puts them in their own bin]. The algorithm is not particularly efficient– it's more n^2 than n log(n) which it should be.

Definition at line 867 of file nnet-example-functions.cc.

References rnnlm::i, and rnnlm::j.

Referenced by CombineDiscriminativeExamples(), and UnitTestSolvePackingProblem().

869  {
870  groups->clear();
871  std::vector<BaseFloat> group_costs;
872  for (size_t i = 0; i < costs.size(); i++) {
873  bool found_group = false;
874  BaseFloat this_cost = costs[i];
875  for (size_t j = 0; j < groups->size(); j++) {
876  if (group_costs[j] + this_cost <= max_cost) {
877  (*groups)[j].push_back(i);
878  group_costs[j] += this_cost;
879  found_group = true;
880  break;
881  }
882  }
883  if (!found_group) { // Put this object in a newly created group.
884  groups->resize(groups->size() + 1);
885  groups->back().push_back(i);
886  group_costs.push_back(this_cost);
887  }
888  }
889 }
float BaseFloat
Definition: kaldi-types.h:29
void SplitDiscriminativeExample ( const SplitDiscriminativeExampleConfig &  config,
const TransitionModel &  tmodel,
const DiscriminativeNnetExample &  eg,
std::vector< DiscriminativeNnetExample > *  egs_out,
SplitExampleStats *  stats_out 
)

Split a "discriminative example" into multiple pieces, splitting where the lattice has "pinch points".

Definition at line 764 of file nnet-example-functions.cc.

References DiscriminativeExampleSplitter::Split().

Referenced by main().

769  {
770  DiscriminativeExampleSplitter splitter(config, tmodel, eg, egs_out);
771  splitter.Split(stats_out);
772 }
BaseFloat TotalNnetTrainingWeight ( const std::vector< NnetExample > &  egs)

Returns the total weight summed over all the examples...

just a simple utility function.

Definition at line 248 of file nnet-update.cc.

References rnnlm::i, and rnnlm::j.

Referenced by DoBackpropSingleThreaded(), main(), DoBackpropParallelClass::operator()(), and NnetExampleBackgroundReader::ReadExamples().

248  {
249  double ans = 0.0;
250  for (size_t i = 0; i < egs.size(); i++)
251  for (size_t j = 0; j < egs[i].labels.size(); j++) // for each labeled frame
252  for (size_t k = 0; k < egs[i].labels[j].size(); k++)
253  ans += egs[i].labels[j][k].second;
254  return ans;
255 }
int64 TrainNnetSimple ( const NnetSimpleTrainerConfig &  config,
Nnet *  nnet,
SequentialNnetExampleReader *  reader,
double *  tot_weight = NULL,
double *  tot_logprob = NULL 
)

Train on all the examples it can read from the reader.

This does training in a single thread, but it uses a separate thread to read in the examples and format the input data on the CPU; this saves us time when using GPUs. Returns the number of examples processed. Outputs to tot_weight and tot_logprob_per_frame, if non-NULL, the total weight of the examples (typically equal to the number of examples) and the total logprob objective function.

Definition at line 147 of file train-nnet.cc.

References DoBackprop(), NnetExampleBackgroundReader::GetNextMinibatch(), rnnlm::i, KALDI_ASSERT, KALDI_LOG, KALDI_WARN, NnetSimpleTrainerConfig::minibatch_size, and NnetSimpleTrainerConfig::minibatches_per_phase.

Referenced by main().

151  {
152  int64 num_egs_processed = 0;
153  double tot_weight = 0.0, tot_logprob = 0.0;
154  NnetExampleBackgroundReader background_reader(config.minibatch_size,
155  nnet, reader);
156  KALDI_ASSERT(config.minibatches_per_phase > 0);
157  while (true) {
158  // Iterate over phases. A phase of training is just a certain number of
159  // minibatches, and its only significance is that it's the periodicity with
160  // which we print diagnostics.
161  double tot_weight_this_phase = 0.0, tot_logprob_this_phase = 0.0;
162 
163  int32 i;
164  for (i = 0; i < config.minibatches_per_phase; i++) {
165  std::vector<NnetExample> examples;
166  Matrix<BaseFloat> examples_formatted;
167  double minibatch_total_weight; // this will normally equal minibatch size.
168  if (!background_reader.GetNextMinibatch(&examples, &examples_formatted,
169  &minibatch_total_weight))
170  break;
171  tot_logprob_this_phase += DoBackprop(*nnet, examples, &examples_formatted,
172  nnet, NULL);
173  tot_weight_this_phase += minibatch_total_weight;
174  num_egs_processed += examples.size();
175  }
176  if (i != 0) {
177  KALDI_LOG << "Training objective function (this phase) is "
178  << (tot_logprob_this_phase / tot_weight_this_phase) << " over "
179  << tot_weight_this_phase << " frames.";
180  }
181  tot_weight += tot_weight_this_phase;
182  tot_logprob += tot_logprob_this_phase;
183  if (i != config.minibatches_per_phase) {
184  // did not get all the minibatches we wanted because no more input.
185  // this is true if and only if we did "break" in the loop over i above.
186  break;
187  }
188  }
189  if (tot_weight == 0.0) {
190  KALDI_WARN << "No data seen.";
191  } else {
192  KALDI_LOG << "Did backprop on " << tot_weight
193  << " examples, average log-prob per frame is "
194  << (tot_logprob / tot_weight);
195  KALDI_LOG << "[this line is to be parsed by a script:] log-prob-per-frame="
196  << (tot_logprob / tot_weight);
197  }
198  if (tot_weight_ptr) *tot_weight_ptr = tot_weight;
199  if (tot_logprob_ptr) *tot_logprob_ptr = tot_logprob;
200  return num_egs_processed;
201 }
double DoBackprop(const Nnet &nnet, const std::vector< NnetExample > &examples, Nnet *nnet_to_update, double *tot_accuracy)
This function computes the objective function and either updates the model or adds to parameter gradi...
Definition: nnet-update.cc:265
#define KALDI_WARN
Definition: kaldi-error.h:130
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
#define KALDI_LOG
Definition: kaldi-error.h:133
void kaldi::nnet2::TransformTrainingExample ( const Matrix< BaseFloat > &  fmllr_mat,
BaseFloat  noise_factor,
NnetExample eg 
)

Definition at line 29 of file nnet-perturb-egs-fmllr.cc.

References MatrixBase< Real >::AddMat(), kaldi::ApplyAffineTransform(), CompressedMatrix::CopyFromMat(), NnetExample::input_frames, MatrixBase< Real >::NumRows(), and MatrixBase< Real >::Scale().

Referenced by main().

31  {
32 
33  Matrix<BaseFloat> input_frames(eg->input_frames);
34  Matrix<BaseFloat> transformed_frames(input_frames);
35 
36  for (int32 t = 0; t < transformed_frames.NumRows(); t++) {
37  SubVector<BaseFloat> row(transformed_frames, t);
38  ApplyAffineTransform(fmllr_mat, &row);
39  }
40  input_frames.Scale(1.0 - noise_factor);
41  input_frames.AddMat(noise_factor, transformed_frames);
42  eg->input_frames.CopyFromMat(input_frames);
43 }
CompressedMatrix input_frames
The input data, with NumRows() >= labels.size() + left_context; it includes features to the left and ...
Definition: nnet-example.h:49
void ApplyAffineTransform(const MatrixBase< BaseFloat > &xform, VectorBase< BaseFloat > *vec)
Applies the affine transform 'xform' to the vector 'vec' and overwrites the contents of 'vec'...
Represents a non-allocating general vector which can be defined as a sub-vector of higher-level vecto...
Definition: kaldi-vector.h:482
void CopyFromMat(const MatrixBase< Real > &mat, CompressionMethod method=kAutomaticMethod)
This will resize *this and copy the contents of mat to *this.
void kaldi::nnet2::UnitTestAdditiveNoiseComponent ( )

Definition at line 434 of file nnet-component-test.cc.

References rnnlm::i, AdditiveNoiseComponent::InitFromString(), KALDI_ERR, KALDI_WARN, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

434  {
435  // We're testing that the gradients are computed correctly:
436  // the input gradients and the model gradients.
437 
438  int32 num_fail = 0, num_tries = 4;
439  for (int32 i = 0; i < num_tries; i++) {
440  try {
441  int32 input_dim = 10 + Rand() % 50;
442  {
443  AdditiveNoiseComponent additive_noise_component(input_dim, 0.1);
444  UnitTestGenericComponentInternal(additive_noise_component);
445  }
446  {
447  AdditiveNoiseComponent additive_noise_component;
448  additive_noise_component.InitFromString("dim=15 stddev=0.2");
449  UnitTestGenericComponentInternal(additive_noise_component);
450  }
451  } catch (...) {
452  KALDI_WARN << "Ignoring failure in AdditiveNoiseComponent test";
453  num_fail++;
454  }
455  }
456  if (num_fail >= num_tries/2) {
457  KALDI_ERR << "Too many test failures.";
458  }
459 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
This is a bit similar to dropout but adding (not multiplying) Gaussian noise with a given standard de...
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_WARN
Definition: kaldi-error.h:130
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestAffineComponent ( )

Definition at line 337 of file nnet-component-test.cc.

References AffineComponent::Init(), AffineComponent::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

337  {
338  BaseFloat learning_rate = 0.01,
339  param_stddev = 0.1, bias_stddev = 1.0;
340  int32 input_dim = 5 + Rand() % 10, output_dim = 5 + Rand() % 10;
341  {
342  AffineComponent component;
343  if (Rand() % 2 == 0) {
344  component.Init(learning_rate, input_dim, output_dim,
345  param_stddev, bias_stddev);
346  } else {
347  Matrix<BaseFloat> mat(output_dim + 1, input_dim);
348  mat.SetRandn();
349  mat.Scale(param_stddev);
350  WriteKaldiObject(mat, "tmpf", true);
351  Sleep(0.5);
352  component.Init(learning_rate, "tmpf");
353  unlink("tmpf");
354  }
356  }
357  {
358  const char *str = "learning-rate=0.01 input-dim=10 output-dim=15 param-stddev=0.1";
359  AffineComponent component;
360  component.InitFromString(str);
362  }
363 }
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev)
void Sleep(float seconds)
Definition: kaldi-utils.cc:45
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void WriteKaldiObject(const C &c, const std::string &filename, bool binary)
Definition: kaldi-io.h:257
void kaldi::nnet2::UnitTestAffineComponentPreconditioned ( )

Definition at line 478 of file nnet-component-test.cc.

References AffineComponentPreconditioned::Init(), AffineComponentPreconditioned::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

478  {
479  BaseFloat learning_rate = 0.01,
480  param_stddev = 0.1, bias_stddev = 1.0, alpha = 0.01,
481  max_change = 100.0;
482  int32 input_dim = 5 + Rand() % 10, output_dim = 5 + Rand() % 10;
483  {
485  if (Rand() % 2 == 0) {
486  component.Init(learning_rate, input_dim, output_dim,
487  param_stddev, bias_stddev,
488  alpha, max_change);
489  } else {
490  Matrix<BaseFloat> mat(output_dim + 1, input_dim);
491  mat.SetRandn();
492  mat.Scale(param_stddev);
493  WriteKaldiObject(mat, "tmpf", true);
494  Sleep(0.5);
495  component.Init(learning_rate, alpha, max_change, "tmpf");
496  unlink("tmpf");
497  }
499  }
500  {
501  const char *str = "learning-rate=0.01 input-dim=16 output-dim=15 param-stddev=0.1 alpha=0.01";
503  component.InitFromString(str);
505  }
506 }
void Sleep(float seconds)
Definition: kaldi-utils.cc:45
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void WriteKaldiObject(const C &c, const std::string &filename, bool binary)
Definition: kaldi-io.h:257
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, BaseFloat alpha, BaseFloat max_change)
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
void kaldi::nnet2::UnitTestAffineComponentPreconditionedOnline ( )

Definition at line 509 of file nnet-component-test.cc.

References AffineComponentPreconditionedOnline::Init(), AffineComponentPreconditionedOnline::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

509  {
510  BaseFloat learning_rate = 0.01,
511  param_stddev = 0.1, bias_stddev = 1.0, num_samples_history = 2000.0, alpha = 4.0,
512  max_change_per_sample = 0.1, update_period = 1;
513  int32 input_dim = 5 + Rand() % 10, output_dim = 5 + Rand() % 10,
514  rank_in = 1 + Rand() % 5, rank_out = 1 + Rand() % 5;
515  {
517  if (Rand() % 2 == 0) {
518  component.Init(learning_rate, input_dim, output_dim,
519  param_stddev, bias_stddev,
520  rank_in, rank_out, update_period,
521  num_samples_history, alpha,
522  max_change_per_sample);
523  } else {
524  Matrix<BaseFloat> mat(output_dim + 1, input_dim);
525  mat.SetRandn();
526  mat.Scale(param_stddev);
527  WriteKaldiObject(mat, "tmpf", true);
528  Sleep(0.5);
529  component.Init(learning_rate, rank_in, rank_out,
530  update_period, num_samples_history, alpha,
531  max_change_per_sample, "tmpf");
532  unlink("tmpf");
533  }
535  }
536  {
537  const char *str = "learning-rate=0.01 input-dim=16 output-dim=15 param-stddev=0.1 num-samples-history=3000 alpha=2.0 update-period=1 rank-in=5 rank-out=6";
539  component.InitFromString(str);
541  }
542 }
void Sleep(float seconds)
Definition: kaldi-utils.cc:45
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 rank_in, int32 rank_out, int32 update_period, BaseFloat num_samples_history, BaseFloat alpha, BaseFloat max_change_per_sample)
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
void UnitTestGenericComponentInternal(const Component &component)
Keywords: natural gradient descent, NG-SGD, naturalgradient.
void WriteKaldiObject(const C &c, const std::string &filename, bool binary)
Definition: kaldi-io.h:257
void kaldi::nnet2::UnitTestAmNnet ( )

Definition at line 29 of file am-nnet-test.cc.

References VectorBase< Real >::ApplyExp(), kaldi::GenRandContextDependencyLarge(), GenRandomNnet(), kaldi::GetDefaultTopology(), rnnlm::i, KALDI_ASSERT, TransitionModel::NumPdfs(), AmNnet::Read(), VectorBase< Real >::Scale(), AmNnet::SetPriors(), VectorBase< Real >::SetRandn(), VectorBase< Real >::Sum(), and AmNnet::Write().

Referenced by main().

29  {
30  std::vector<int32> phones;
31  phones.push_back(1);
32  for (int32 i = 2; i < 20; i++)
33  if (rand() % 2 == 0)
34  phones.push_back(i);
35  int32 N = 2 + rand() % 2, // context-size N is 2 or 3.
36  P = rand() % N; // Central-phone is random on [0, N)
37 
38  std::vector<int32> num_pdf_classes;
39 
40  ContextDependency *ctx_dep =
41  GenRandContextDependencyLarge(phones, N, P,
42  true, &num_pdf_classes);
43 
44  HmmTopology topo = GetDefaultTopology(phones);
45 
46  TransitionModel trans_model(*ctx_dep, topo);
47 
48  delete ctx_dep; // We won't need this further.
49  ctx_dep = NULL;
50 
51  int32 input_dim = 40, output_dim = trans_model.NumPdfs();
52  Nnet *nnet = GenRandomNnet(input_dim, output_dim);
53 
54  AmNnet am_nnet(*nnet);
55  delete nnet;
56  nnet = NULL;
57  Vector<BaseFloat> priors(output_dim);
58  priors.SetRandn();
59  priors.ApplyExp();
60  priors.Scale(1.0 / priors.Sum());
61 
62  am_nnet.SetPriors(priors);
63 
64  bool binary = (rand() % 2 == 0);
65  std::ostringstream os;
66  am_nnet.Write(os, binary);
67  AmNnet am_nnet2;
68  std::istringstream is(os.str());
69  am_nnet2.Read(is, binary);
70 
71  std::ostringstream os2;
72  am_nnet2.Write(os2, binary);
73 
74  KALDI_ASSERT(os2.str() == os.str());
75 }
HmmTopology GetDefaultTopology(const std::vector< int32 > &phones_in)
This function returns a HmmTopology object giving a normal 3-state topology, covering all phones in t...
ContextDependency * GenRandContextDependencyLarge(const std::vector< int32 > &phone_ids, int N, int P, bool ensure_all_covered, std::vector< int32 > *hmm_lengths)
GenRandContextDependencyLarge is like GenRandContextDependency but generates a larger tree with speci...
Definition: context-dep.cc:97
A class for storing topology information for phones.
Definition: hmm-topology.h:94
Nnet * GenRandomNnet(int32 input_dim, int32 output_dim)
This function generates a random neural net, for testing purposes.
Definition: nnet-nnet.cc:772
virtual int32 NumPdfs() const
NumPdfs() returns the number of acoustic pdfs (they are numbered 0.. NumPdfs()-1).
Definition: context-dep.h:71
void Read(std::istream &is, bool binary)
Definition: am-nnet.cc:39
void Write(std::ostream &os, bool binary) const
Definition: am-nnet.cc:31
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void kaldi::nnet2::UnitTestBlockAffineComponent ( )

Definition at line 544 of file nnet-component-test.cc.

References BlockAffineComponent::Init(), BlockAffineComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

544  {
545  BaseFloat learning_rate = 0.01,
546  param_stddev = 0.1, bias_stddev = 0.1;
547  int32 num_blocks = 1 + Rand() % 3,
548  input_dim = num_blocks * (2 + Rand() % 4),
549  output_dim = num_blocks * (2 + Rand() % 4);
550 
551  {
552  BlockAffineComponent component;
553  component.Init(learning_rate, input_dim, output_dim,
554  param_stddev, bias_stddev, num_blocks);
556  }
557  {
558  const char *str = "learning-rate=0.01 input-dim=10 output-dim=15 param-stddev=0.1 num-blocks=5";
559  BlockAffineComponent component;
560  component.InitFromString(str);
562  }
563 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 num_blocks)
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestBlockAffineComponentPreconditioned ( )

Definition at line 565 of file nnet-component-test.cc.

References BlockAffineComponentPreconditioned::Init(), BlockAffineComponentPreconditioned::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

565  {
566  BaseFloat learning_rate = 0.01,
567  param_stddev = 0.1, bias_stddev = 1.0, alpha = 3.0;
568  int32 num_blocks = 1 + Rand() % 3,
569  input_dim = num_blocks * (2 + Rand() % 4),
570  output_dim = num_blocks * (2 + Rand() % 4);
571 
572  {
574  component.Init(learning_rate, input_dim, output_dim,
575  param_stddev, bias_stddev, num_blocks, alpha);
577  }
578  {
579  const char *str = "learning-rate=0.01 input-dim=10 output-dim=15 param-stddev=0.1 num-blocks=5 alpha=3.0";
581  component.InitFromString(str);
583  }
584 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, BaseFloat param_stddev, BaseFloat bias_stddev, int32 num_blocks, BaseFloat alpha)
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestConvolutional1dComponent ( )

Definition at line 365 of file nnet-component-test.cc.

References Convolutional1dComponent::Init(), Convolutional1dComponent::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

365  {
366  BaseFloat learning_rate = 0.01,
367  param_stddev = 0.1, bias_stddev = 1.0;
368  int32 patch_stride = 10, patch_step = 1, patch_dim = 4;
369  int32 num_patches = 1 + (patch_stride - patch_dim) / patch_step;
370  int32 num_splice = 5 + Rand() % 10, num_filters = 5 + Rand() % 10;
371  int32 input_dim = patch_stride * num_splice;
372  int32 filter_dim = patch_dim * num_splice;
373  int32 output_dim = num_patches * num_filters;
374  {
375  Convolutional1dComponent component;
376  if (Rand() % 2 == 0) {
377  component.Init(learning_rate, input_dim, output_dim,
378  patch_dim, patch_step, patch_stride,
379  param_stddev, bias_stddev, true);
380  } else {
381  Matrix<BaseFloat> mat(num_filters, filter_dim + 1);
382  mat.SetRandn();
383  mat.Scale(param_stddev);
384  WriteKaldiObject(mat, "tmpf", true);
385  Sleep(0.5);
386  component.Init(learning_rate, patch_dim,
387  patch_step, patch_stride, "tmpf", false);
388  unlink("tmpf");
389  }
391  }
392  {
393  // appended-conv is false by default
394  const char *str = "learning-rate=0.01 input-dim=100 output-dim=70 param-stddev=0.1 patch-dim=4 patch-step=1 patch-stride=10";
395  Convolutional1dComponent component;
396  component.InitFromString(str);
398  }
399  {
400  const char *str = "learning-rate=0.01 input-dim=100 output-dim=70 param-stddev=0.1 patch-dim=4 patch-step=1 patch-stride=10 appended-conv=true";
401  Convolutional1dComponent component;
402  component.InitFromString(str);
404  }
405 }
void Sleep(float seconds)
Definition: kaldi-utils.cc:45
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void Init(BaseFloat learning_rate, int32 input_dim, int32 output_dim, int32 patch_dim, int32 patch_step, int32 patch_stride, BaseFloat param_stddev, BaseFloat bias_stddev, bool appended_conv)
void WriteKaldiObject(const C &c, const std::string &filename, bool binary)
Definition: kaldi-io.h:257
void InitFromString(std::string args)
Initialize, typically from a line of a config file.
Convolutional1dComponent implements convolution over frequency axis.
void kaldi::nnet2::UnitTestDctComponent ( )

Definition at line 607 of file nnet-component-test.cc.

References DctComponent::Init(), DctComponent::InitFromString(), rnnlm::n, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

607  {
608  int32 m = 1 + Rand() % 4, n = 1 + Rand() % 4,
609  dct_dim = m, dim = m * n;
610  bool reorder = (Rand() % 2 == 0);
611  {
612  DctComponent component;
613  component.Init(dim, dct_dim, reorder);
615  }
616  {
617  const char *str = "dim=10 dct-dim=5 reorder=true";
618  DctComponent component;
619  component.InitFromString(str);
621  }
622  {
623  const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=1";
624  DctComponent component;
625  component.InitFromString(str);
627  }
628  {
629  const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=2";
630  DctComponent component;
631  component.InitFromString(str);
633  }
634  {
635  const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=3";
636  DctComponent component;
637  component.InitFromString(str);
639  }
640  {
641  const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=4";
642  DctComponent component;
643  component.InitFromString(str);
645  }
646 }
void Init(int32 dim, int32 dct_dim, bool reorder, int32 keep_dct_dim=0)
Discrete cosine transform.
struct rnnlm::@11::@12 n
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestDropoutComponent ( )

Definition at line 407 of file nnet-component-test.cc.

References rnnlm::i, DropoutComponent::InitFromString(), KALDI_ERR, KALDI_WARN, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

407  {
408  // We're testing that the gradients are computed correctly:
409  // the input gradients and the model gradients.
410 
411  int32 num_fail = 0, num_tries = 4;
412  for (int32 i = 0; i < num_tries; i++) {
413  try {
414  int32 input_dim = 10 + Rand() % 50;
415  {
416  DropoutComponent dropout_component(input_dim, 0.5, 0.3);
417  UnitTestGenericComponentInternal(dropout_component);
418  }
419  {
420  DropoutComponent dropout_component;
421  dropout_component.InitFromString("dim=15 dropout-proportion=0.6 dropout-scale=0.1");
422  UnitTestGenericComponentInternal(dropout_component);
423  }
424  } catch (...) {
425  KALDI_WARN << "Ignoring test failure in UnitTestDropoutComponent().";
426  num_fail++;
427  }
428  }
429  if (num_fail >= num_tries/2) {
430  KALDI_ERR << "Too many test failures.";
431  }
432 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two...
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_WARN
Definition: kaldi-error.h:130
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestFixedAffineComponent ( )

Definition at line 661 of file nnet-component-test.cc.

References FixedAffineComponent::Init(), rnnlm::n, kaldi::Rand(), CuMatrixBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

661  {
662  int32 m = 15 + Rand() % 4, n = 15 + Rand() % 4;
663  {
664  CuMatrix<BaseFloat> mat(m, n);
665  mat.SetRandn();
666  FixedAffineComponent component;
667  component.Init(mat);
669  }
670 }
FixedAffineComponent is an affine transform that is supplied at network initialization time and is no...
void Init(const CuMatrixBase< BaseFloat > &matrix)
matrix should be of size input-dim+1 to output-dim, last col is offset
struct rnnlm::@11::@12 n
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestFixedBiasComponent ( )

Definition at line 683 of file nnet-component-test.cc.

References FixedBiasComponent::Init(), kaldi::Rand(), CuVectorBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

683  {
684  int32 m = 1 + Rand() % 20;
685  {
686  CuVector<BaseFloat> vec(m);
687  vec.SetRandn();
688  FixedBiasComponent component;
689  component.Init(vec);
691  }
692 }
void Init(const CuVectorBase< BaseFloat > &scales)
FixedBiasComponent applies a fixed per-element bias; it's similar to the AddShift component in the nn...
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestFixedLinearComponent ( )

Definition at line 649 of file nnet-component-test.cc.

References FixedLinearComponent::Init(), rnnlm::n, kaldi::Rand(), CuMatrixBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

649  {
650  int32 m = 1 + Rand() % 4, n = 1 + Rand() % 4;
651  {
652  CuMatrix<BaseFloat> mat(m, n);
653  mat.SetRandn();
654  FixedLinearComponent component;
655  component.Init(mat);
657  }
658 }
void Init(const CuMatrixBase< BaseFloat > &matrix)
struct rnnlm::@11::@12 n
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
FixedLinearComponent is a linear transform that is supplied at network initialization time and is not...
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestFixedScaleComponent ( )

Definition at line 672 of file nnet-component-test.cc.

References FixedScaleComponent::Init(), kaldi::Rand(), CuVectorBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

672  {
673  int32 m = 1 + Rand() % 20;
674  {
675  CuVector<BaseFloat> vec(m);
676  vec.SetRandn();
677  FixedScaleComponent component;
678  component.Init(vec);
680  }
681 }
FixedScaleComponent applies a fixed per-element scale; it's similar to the Rescale component in the n...
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void Init(const CuVectorBase< BaseFloat > &scales)
void kaldi::nnet2::UnitTestGenericComponent ( std::string  extra_str = "")

Definition at line 244 of file nnet-component-test.cc.

References kaldi::Rand(), and UnitTestGenericComponentInternal().

244  {
245  // works if it has an initializer from int,
246  // e.g. tanh, sigmoid.
247 
248  // We're testing that the gradients are computed correctly:
249  // the input gradients and the model gradients.
250 
251  int32 input_dim = 10 + Rand() % 50;
252  {
253  T component(input_dim);
255  }
256  {
257  T component;
258  component.InitFromString(static_cast<std::string>("dim=15 ") + extra_str);
260  }
261 }
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestGenericComponentInternal ( const Component component,
const ChunkInfo  in_info,
const ChunkInfo  out_info 
)

Definition at line 28 of file nnet-component-test.cc.

References CuVectorBase< Real >::AddMatVec(), Component::Backprop(), Component::BackpropNeedsInput(), Component::BackpropNeedsOutput(), Component::Copy(), UpdatableComponent::DotProduct(), rnnlm::i, KALDI_ASSERT, KALDI_ERR, KALDI_LOG, KALDI_WARN, kaldi::kNoTrans, kaldi::kTrans, ChunkInfo::NumCols(), ChunkInfo::NumRows(), UpdatableComponent::PerturbParams(), Component::Propagate(), kaldi::Rand(), Component::ReadNew(), RandomComponent::ResetGenerator(), CuMatrixBase< Real >::Row(), CuVectorBase< Real >::SetRandn(), CuMatrixBase< Real >::SetRandn(), Output::Stream(), Input::Stream(), kaldi::TraceMatMat(), and Component::Write().

Referenced by UnitTestAdditiveNoiseComponent(), UnitTestAffineComponent(), UnitTestAffineComponentPreconditioned(), UnitTestAffineComponentPreconditionedOnline(), UnitTestBlockAffineComponent(), UnitTestBlockAffineComponentPreconditioned(), UnitTestConvolutional1dComponent(), UnitTestDctComponent(), UnitTestDropoutComponent(), UnitTestFixedAffineComponent(), UnitTestFixedBiasComponent(), UnitTestFixedLinearComponent(), UnitTestFixedScaleComponent(), UnitTestGenericComponent(), UnitTestGenericComponentInternal(), UnitTestMaxoutComponent(), UnitTestMaxpoolingComponent(), UnitTestPnormComponent(), UnitTestScaleComponent(), UnitTestSigmoidComponent(), UnitTestSpliceComponent(), and UnitTestSumGroupComponent().

30  {
31 
32  CuMatrix<BaseFloat> input(in_info.NumRows(), in_info.NumCols()),
33  output(1, out_info.NumRows() * out_info.NumCols());
34  input.SetRandn();
35  CuVector<BaseFloat> objf_vec(out_info.NumCols()); // objective function is linear function of output.
36  objf_vec.SetRandn(); // set to Gaussian noise.
37 
38  int32 rand_seed = Rand();
39 
40  RandomComponent *rand_component =
41  const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
42  if (rand_component != NULL) {
43  srand(rand_seed);
44  rand_component->ResetGenerator();
45  }
46  component.Propagate(in_info, out_info, input, &output);
47  {
48  bool binary = (Rand() % 2 == 0);
49  Output ko("tmpf", binary);
50  component.Write(ko.Stream(), binary);
51  }
52  Component *component_copy;
53  {
54  bool binary_in;
55  Input ki("tmpf", &binary_in);
56  component_copy = Component::ReadNew(ki.Stream(), binary_in);
57  }
58  unlink("tmpf");
59 
60  { // Test backward derivative is correct.
61  CuVector<BaseFloat> output_objfs(out_info.NumRows());
62  output_objfs.AddMatVec(1.0, output, kNoTrans, objf_vec, 0.0);
63  BaseFloat objf = output_objfs.Sum();
64 
65 
66  CuMatrix<BaseFloat> output_deriv(output.NumRows(), output.NumCols());
67  for (int32 i = 0; i < output_deriv.NumRows(); i++)
68  output_deriv.Row(i).CopyFromVec(objf_vec);
69 
70  CuMatrix<BaseFloat> input_deriv(input.NumRows(), input.NumCols());
71 
72 
73  CuMatrix<BaseFloat> empty_mat;
74  CuMatrix<BaseFloat> &input_ref =
75  (component_copy->BackpropNeedsInput() ? input : empty_mat),
76  &output_ref =
77  (component_copy->BackpropNeedsOutput() ? output : empty_mat);
78 
79  component_copy->Backprop(in_info, out_info, input_ref, output_ref,
80  output_deriv, NULL, &input_deriv);
81 
82  int32 num_ok = 0, num_bad = 0, num_tries = 10;
83  KALDI_LOG << "Comparing feature gradients " << num_tries << " times.";
84  for (int32 i = 0; i < num_tries; i++) {
85  CuMatrix<BaseFloat> perturbed_input(input.NumRows(), input.NumCols());
86  {
87  RandomComponent *rand_component =
88  const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
89  if (rand_component != NULL) {
90  srand(rand_seed);
91  rand_component->ResetGenerator();
92  }
93  }
94  perturbed_input.SetRandn();
95  perturbed_input.Scale(1.0e-04); // scale by a small amount so it's like a delta.
96  BaseFloat predicted_difference = TraceMatMat(perturbed_input,
97  input_deriv, kTrans);
98  perturbed_input.AddMat(1.0, input); // now it's the input + a delta.
99  { // Compute objf with perturbed input and make sure it matches
100  // prediction.
101  CuMatrix<BaseFloat> perturbed_output(output.NumRows(), output.NumCols());
102  {
103  RandomComponent *rand_component =
104  const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
105  if (rand_component != NULL) {
106  srand(rand_seed);
107  rand_component->ResetGenerator();
108  }
109  }
110  component.Propagate(in_info, out_info, perturbed_input, &perturbed_output);
111  CuVector<BaseFloat> perturbed_output_objfs(out_info.NumRows());
112  perturbed_output_objfs.AddMatVec(1.0, perturbed_output, kNoTrans,
113  objf_vec, 0.0);
114  BaseFloat perturbed_objf = perturbed_output_objfs.Sum(),
115  observed_difference = perturbed_objf - objf;
116  KALDI_LOG << "Input gradients: comparing " << predicted_difference
117  << " and " << observed_difference;
118  if (fabs(predicted_difference - observed_difference) >
119  0.15 * fabs((predicted_difference + observed_difference)/2) &&
120  fabs(predicted_difference - observed_difference) > 1.0e-06) {
121  KALDI_WARN << "Bad difference!";
122  num_bad++;
123  } else {
124  num_ok++;
125  }
126  }
127  }
128  KALDI_LOG << "Succeeded for " << num_ok << " out of " << num_tries
129  << " tries.";
130  if (num_ok <= num_bad) {
131  delete component_copy;
132  KALDI_ERR << "Feature-derivative check failed";
133  }
134  }
135 
136  UpdatableComponent *ucomponent =
137  dynamic_cast<UpdatableComponent*>(component_copy);
138 
139  if (ucomponent != NULL) { // Test parameter derivative is correct.
140 
141  int32 num_ok = 0, num_bad = 0, num_tries = 10;
142  KALDI_LOG << "Comparing model gradients " << num_tries << " times.";
143  for (int32 i = 0; i < num_tries; i++) {
144  UpdatableComponent *perturbed_ucomponent =
145  dynamic_cast<UpdatableComponent*>(ucomponent->Copy()),
146  *gradient_ucomponent =
147  dynamic_cast<UpdatableComponent*>(ucomponent->Copy());
148  KALDI_ASSERT(perturbed_ucomponent != NULL);
149  gradient_ucomponent->SetZero(true); // set params to zero and treat as gradient.
150  BaseFloat perturb_stddev = 5.0e-04;
151  perturbed_ucomponent->PerturbParams(perturb_stddev);
152 
153  CuVector<BaseFloat> output_objfs(out_info.NumRows());
154  output_objfs.AddMatVec(1.0, output, kNoTrans, objf_vec, 0.0);
155  BaseFloat objf = output_objfs.Sum();
156 
157  CuMatrix<BaseFloat> output_deriv(output.NumRows(), output.NumCols());
158  for (int32 i = 0; i < output_deriv.NumRows(); i++)
159  output_deriv.Row(i).CopyFromVec(objf_vec);
160  CuMatrix<BaseFloat> input_deriv; // (input.NumRows(), input.NumCols());
161 
162  // This will compute the parameter gradient.
163  ucomponent->Backprop(in_info, out_info, input, output, output_deriv,
164  gradient_ucomponent, &input_deriv);
165 
166  // Now compute the perturbed objf.
167  BaseFloat objf_perturbed;
168  {
169  CuMatrix<BaseFloat> output_perturbed; // (num_egs, output_dim);
170  {
171  RandomComponent *rand_component =
172  const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
173  if (rand_component != NULL) {
174  srand(rand_seed);
175  rand_component->ResetGenerator();
176  }
177  }
178  perturbed_ucomponent->Propagate(in_info, out_info, input, &output_perturbed);
179  CuVector<BaseFloat> output_objfs_perturbed(out_info.NumRows());
180  output_objfs_perturbed.AddMatVec(1.0, output_perturbed,
181  kNoTrans, objf_vec, 0.0);
182  objf_perturbed = output_objfs_perturbed.Sum();
183  }
184 
185  BaseFloat delta_objf_observed = objf_perturbed - objf,
186  delta_objf_predicted = (perturbed_ucomponent->DotProduct(*gradient_ucomponent) -
187  ucomponent->DotProduct(*gradient_ucomponent));
188 
189  KALDI_LOG << "Model gradients: comparing " << delta_objf_observed
190  << " and " << delta_objf_predicted;
191  if (fabs(delta_objf_predicted - delta_objf_observed) >
192  0.05 * (fabs(delta_objf_predicted + delta_objf_observed)/2) &&
193  fabs(delta_objf_predicted - delta_objf_observed) > 1.0e-06) {
194  KALDI_WARN << "Bad difference!";
195  num_bad++;
196  } else {
197  num_ok++;
198  }
199  delete perturbed_ucomponent;
200  delete gradient_ucomponent;
201  }
202  if (num_ok < num_bad) {
203  delete component_copy;
204  KALDI_ERR << "model-derivative check failed";
205  }
206  }
207  delete component_copy; // No longer needed.
208 }
virtual bool BackpropNeedsInput() const
Abstract class, basic element of the network, it is a box with defined inputs, outputs, and tranformation functions interface.
virtual bool BackpropNeedsOutput() const
const CuSubVector< Real > Row(MatrixIndexT i) const
Definition: cu-matrix.h:545
virtual Component * Copy() const =0
Copy component (deep copy).
int32 NumRows() const
Returns the number of rows that we expect the feature matrix to have.
float BaseFloat
Definition: kaldi-types.h:29
virtual BaseFloat DotProduct(const UpdatableComponent &other) const =0
Here, "other" is a component of the same specific type.
int32 NumCols() const
Returns the number of columns that we expect the feature matrix to have.
virtual void Backprop(const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in_value, const CuMatrixBase< BaseFloat > &out_value, const CuMatrixBase< BaseFloat > &out_deriv, Component *to_update, CuMatrix< BaseFloat > *in_deriv) const =0
Perform backward pass propagation of the derivative, and also either update the model (if to_update =...
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_WARN
Definition: kaldi-error.h:130
Real TraceMatMat(const MatrixBase< Real > &A, const MatrixBase< Real > &B, MatrixTransposeType trans)
We need to declare this here as it will be a friend function.
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
virtual void PerturbParams(BaseFloat stddev)=0
We introduce a new virtual function that only applies to class UpdatableComponent.
virtual void Propagate(const ChunkInfo &in_info, const ChunkInfo &out_info, const CuMatrixBase< BaseFloat > &in, CuMatrixBase< BaseFloat > *out) const =0
Perform forward pass propagation Input->Output.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void AddMatVec(const Real alpha, const CuMatrixBase< Real > &M, MatrixTransposeType trans, const CuVectorBase< Real > &v, const Real beta)
Definition: cu-vector.cc:439
virtual void Write(std::ostream &os, bool binary) const =0
Write component to stream.
#define KALDI_LOG
Definition: kaldi-error.h:133
Class UpdatableComponent is a Component which has trainable parameters and contains some global param...
void kaldi::nnet2::UnitTestGenericComponentInternal ( const Component component)

Definition at line 210 of file nnet-component-test.cc.

References Component::Info(), Component::InputDim(), KALDI_LOG, Component::OutputDim(), kaldi::Rand(), and UnitTestGenericComponentInternal().

210  {
211  int32 input_dim = component.InputDim(),
212  output_dim = component.OutputDim();
213 
214  KALDI_LOG << component.Info();
215  int32 num_egs = 10 + Rand() % 5;
216  int32 num_chunks = 1,
217  first_offset = 0,
218  last_offset = num_egs-1;
219 
220  ChunkInfo in_info(input_dim, num_chunks, first_offset, last_offset);
221  ChunkInfo out_info(output_dim, num_chunks, first_offset, last_offset);
222  UnitTestGenericComponentInternal(component, in_info, out_info);
223 }
virtual int32 InputDim() const =0
Get size of input vectors.
virtual int32 OutputDim() const =0
Get size of output vectors.
virtual std::string Info() const
ChunkInfo is a class whose purpose is to describe the structure of matrices holding features...
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
#define KALDI_LOG
Definition: kaldi-error.h:133
void kaldi::nnet2::UnitTestMaxoutComponent ( )

Definition at line 263 of file nnet-component-test.cc.

References rnnlm::i, MaxoutComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

263  {
264  // works if it has an initializer from int,
265  // e.g. tanh, sigmoid.
266 
267  // We're testing that the gradients are computed correctly:
268  // the input gradients and the model gradients.
269 
270  for (int32 i = 0; i < 5; i++) {
271  int32 output_dim = 10 + Rand() % 20,
272  group_size = 1 + Rand() % 10,
273  input_dim = output_dim * group_size;
274 
275  MaxoutComponent component(input_dim, output_dim);
277  }
278 
279  {
280  MaxoutComponent component;
281  component.InitFromString("input-dim=15 output-dim=5");
283  }
284 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestMaxpoolingComponent ( )

Definition at line 310 of file nnet-component-test.cc.

References rnnlm::i, MaxpoolingComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

310  {
311  // works if it has an initializer from int,
312  // e.g. tanh, sigmoid.
313  // We're testing that the gradients are computed correctly:
314  // the input gradients and the model gradients.
315 
316  for (int32 i = 0; i < 5; i++) {
317  int32 pool_stride = 5 + Rand() % 10,
318  pool_size = 2 + Rand() % 3,
319  num_pools = 1 + Rand() % 10;
320  int32 output_dim = num_pools * pool_stride;
321  int32 num_patches = num_pools * pool_size;
322  int32 input_dim = pool_stride * num_patches;
323 
324  MaxpoolingComponent component(input_dim, output_dim,
325  pool_size, pool_stride);
327  }
328 
329  {
330  MaxpoolingComponent component;
331  component.InitFromString("input-dim=192 output-dim=64 pool-size=3 pool-stride=16");
333  }
334 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
MaxPoolingComponent : Maxpooling component was firstly used in ConvNet for selecting an representativ...
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestNnet ( )

Definition at line 26 of file nnet-nnet-test.cc.

References GenRandomNnet(), KALDI_ASSERT, Nnet::Read(), and Nnet::Write().

Referenced by main().

26  {
27  int32 input_dim = 40, output_dim = 500;
28  Nnet *nnet = GenRandomNnet(input_dim, output_dim);
29 
30  bool binary = (rand() % 2 == 0);
31  std::ostringstream os;
32  nnet->Write(os, binary);
33  Nnet nnet2;
34  std::istringstream is(os.str());
35  nnet2.Read(is, binary);
36 
37  std::ostringstream os2;
38  nnet2.Write(os2, binary);
39 
40  KALDI_ASSERT(os2.str() == os.str());
41  delete nnet;
42 }
void Read(std::istream &is, bool binary)
Definition: nnet-nnet.cc:175
Nnet * GenRandomNnet(int32 input_dim, int32 output_dim)
This function generates a random neural net, for testing purposes.
Definition: nnet-nnet.cc:772
void Write(std::ostream &os, bool binary) const
Definition: nnet-nnet.cc:160
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void kaldi::nnet2::UnitTestNnetCompute ( )

Definition at line 29 of file nnet-compute-test.cc.

References kaldi::AssertEqual(), NnetOnlineComputer::Compute(), NnetOnlineComputer::Flush(), GenRandomNnet(), rnnlm::i, Nnet::Info(), KALDI_LOG, Nnet::LeftContext(), NnetComputation(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), Nnet::RightContext(), and CuMatrixBase< Real >::SetRandn().

Referenced by main().

29  {
30  int32 input_dim = 10 + rand() % 40, output_dim = 100 + rand() % 500;
31  bool pad_input = (rand() % 2 == 0);
32 
33  Nnet *nnet = GenRandomNnet(input_dim, output_dim);
34  KALDI_LOG << "Left context = " << nnet->LeftContext() << ", right context = "
35  << nnet->RightContext() << ", pad-input = " << pad_input;
36  KALDI_LOG << "NNet info is " << nnet->Info();
37  int32 num_feats = 5 + rand() % 1000;
38  CuMatrix<BaseFloat> input(num_feats, input_dim);
39  input.SetRandn();
40 
41  int32 num_output_rows = num_feats -
42  (pad_input ? 0 : nnet->LeftContext() + nnet->RightContext());
43  if (num_output_rows <= 0)
44  return;
45  CuMatrix<BaseFloat> output1(num_output_rows, output_dim);
46  NnetComputation(*nnet, input, pad_input, &output1);
47 
48  CuMatrix<BaseFloat> output2(output1.NumRows(), output1.NumCols());
49  int32 cur_input_pos = 0, cur_output_pos = 0;
50 
51  NnetOnlineComputer computer(*nnet, pad_input);
52  while (cur_input_pos <= num_feats) {
53  int32 feats_left = num_feats - cur_input_pos;
54  CuMatrix<BaseFloat> output_part;
55  if (feats_left > 0) {
56  int32 chunk_size = std::min<int32>(1 + rand() % 10, feats_left);
57  CuSubMatrix<BaseFloat> input_part(input, cur_input_pos, chunk_size,
58  0, input_dim);
59  computer.Compute(input_part, &output_part);
60  cur_input_pos += chunk_size;
61  } else {
62  computer.Flush(&output_part);
63  cur_input_pos++; // will terminate the loop.
64  }
65  if (output_part.NumRows() != 0) {
66  output2.Range(cur_output_pos, output_part.NumRows(),
67  0, output_dim).CopyFromMat(output_part);
68  cur_output_pos += output_part.NumRows();
69  }
70  }
71  AssertEqual(output1, output2);
72  for (int32 i = 0; i < output1.NumRows(); i++) {
73  // just double-check that the frames near the end are right, in case
74  // the test above somehow passed despite that.
75  if (i < 10 || output1.NumRows() - i < 10) {
76  CuSubVector<BaseFloat> vec1(output1, i), vec2(output2, i);
77  AssertEqual(vec1, vec2);
78  }
79  }
80  KALDI_LOG << "OK";
81  delete nnet;
82 }
Nnet * GenRandomNnet(int32 input_dim, int32 output_dim)
This function generates a random neural net, for testing purposes.
Definition: nnet-nnet.cc:772
This class is used for a piece of a CuMatrix.
Definition: matrix-common.h:68
int32 LeftContext() const
Returns the left-context summed over all the Components...
Definition: nnet-nnet.cc:42
MatrixIndexT NumRows() const
Dimensions.
Definition: cu-matrix.h:195
static void AssertEqual(float a, float b, float relative_tolerance=0.001)
assert abs(a - b) <= relative_tolerance * (abs(a)+abs(b))
Definition: kaldi-math.h:273
int32 RightContext() const
Returns the right-context summed over all the Components...
Definition: nnet-nnet.cc:56
void NnetComputation(const Nnet &nnet, const CuMatrixBase< BaseFloat > &input, bool pad_input, CuMatrixBase< BaseFloat > *output)
Does the basic neural net computation, on a sequence of data (e.g.
#define KALDI_LOG
Definition: kaldi-error.h:133
std::string Info() const
Definition: nnet-nnet.cc:257
void kaldi::nnet2::UnitTestNnetComputeChunked ( )

Definition at line 84 of file nnet-compute-test.cc.

References kaldi::AssertEqual(), GenRandomNnet(), rnnlm::i, Nnet::Info(), KALDI_LOG, Nnet::LeftContext(), NnetComputation(), NnetComputationChunked(), MatrixBase< Real >::NumRows(), Nnet::RightContext(), and CuMatrixBase< Real >::SetRandn().

Referenced by main().

84  {
85  int32 input_dim = 10 + rand() % 40, output_dim = 100 + rand() % 500;
86  bool pad_input = true;
87 
88  Nnet *nnet = GenRandomNnet(input_dim, output_dim);
89  int32 num_feats = 100 + rand() % 500;
90  int32 chunk_size = num_feats / (2 + rand() % 10);
91  CuMatrix<BaseFloat> input(num_feats, input_dim);
92  input.SetRandn();
93 
94  KALDI_LOG << "Left context = " << nnet->LeftContext()
95  << ", right context = " << nnet->RightContext()
96  << ", chunk size = " << chunk_size;
97  KALDI_LOG << "NNet info is " << nnet->Info();
98 
99  int32 num_output_rows = num_feats;
100  CuMatrix<BaseFloat> cu_output1(num_output_rows, output_dim);
101  Matrix<BaseFloat> output2(num_output_rows, output_dim);
102  NnetComputation(*nnet, input, pad_input, &cu_output1);
103  NnetComputationChunked(*nnet, Matrix<BaseFloat>(input), chunk_size,
104  &output2);
105  Matrix<BaseFloat> output1(cu_output1);
106  AssertEqual(output1, output2);
107  for (int32 i = 0; i < output1.NumRows(); i++) {
108  // just double-check that the frames near the end are right, in case
109  // the test above somehow passed despite that.
110  if (i < 10 || output1.NumRows() - i < 10) {
111  SubVector<BaseFloat> vec1(output1, i), vec2(output2, i);
112  AssertEqual(vec1, vec2);
113  }
114  }
115  KALDI_LOG << "OK";
116  delete nnet;
117 }
Nnet * GenRandomNnet(int32 input_dim, int32 output_dim)
This function generates a random neural net, for testing purposes.
Definition: nnet-nnet.cc:772
int32 LeftContext() const
Returns the left-context summed over all the Components...
Definition: nnet-nnet.cc:42
static void AssertEqual(float a, float b, float relative_tolerance=0.001)
assert abs(a - b) <= relative_tolerance * (abs(a)+abs(b))
Definition: kaldi-math.h:273
void NnetComputationChunked(const Nnet &nnet, const Matrix< BaseFloat > &input, int32 chunk_size, Matrix< BaseFloat > *output)
Does the basic neural net computation, on a sequence of data (e.g.
int32 RightContext() const
Returns the right-context summed over all the Components...
Definition: nnet-nnet.cc:56
void NnetComputation(const Nnet &nnet, const CuMatrixBase< BaseFloat > &input, bool pad_input, CuMatrixBase< BaseFloat > *output)
Does the basic neural net computation, on a sequence of data (e.g.
#define KALDI_LOG
Definition: kaldi-error.h:133
Represents a non-allocating general vector which can be defined as a sub-vector of higher-level vecto...
Definition: kaldi-vector.h:482
std::string Info() const
Definition: nnet-nnet.cc:257
void kaldi::nnet2::UnitTestNnetDecodable ( )

Definition at line 31 of file online-nnet2-decodable-test.cc.

References DecodableNnet2OnlineOptions::acoustic_scale, VectorBase< Real >::ApplyExp(), kaldi::ApproxEqual(), kaldi::GenRandContextDependencyLarge(), GenRandomNnet(), kaldi::GetDefaultTopology(), rnnlm::i, KALDI_ASSERT, DecodableNnet2Online::LogLikelihood(), DecodableAmNnet::LogLikelihood(), DecodableNnet2OnlineOptions::max_nnet_batch_size, DecodableNnet2Online::NumFramesReady(), DecodableAmNnet::NumFramesReady(), TransitionModel::NumPdfs(), TransitionModel::NumTransitionIds(), DecodableNnet2OnlineOptions::pad_input, VectorBase< Real >::Scale(), AmNnet::SetPriors(), VectorBase< Real >::SetRandn(), MatrixBase< Real >::SetRandn(), and VectorBase< Real >::Sum().

Referenced by main().

31  {
32  std::vector<int32> phones;
33  phones.push_back(1);
34  for (int32 i = 2; i < 20; i++)
35  if (rand() % 2 == 0)
36  phones.push_back(i);
37  int32 N = 2 + rand() % 2, // context-size N is 2 or 3.
38  P = rand() % N; // Central-phone is random on [0, N)
39 
40  std::vector<int32> num_pdf_classes;
41 
42  ContextDependency *ctx_dep =
43  GenRandContextDependencyLarge(phones, N, P,
44  true, &num_pdf_classes);
45 
46  HmmTopology topo = GetDefaultTopology(phones);
47 
48  TransitionModel trans_model(*ctx_dep, topo);
49 
50  delete ctx_dep; // We won't need this further.
51  ctx_dep = NULL;
52 
53  int32 input_dim = 40, output_dim = trans_model.NumPdfs();
54  Nnet *nnet = GenRandomNnet(input_dim, output_dim);
55 
56  AmNnet am_nnet(*nnet);
57  delete nnet;
58  nnet = NULL;
59  Vector<BaseFloat> priors(output_dim);
60  priors.SetRandn();
61  priors.ApplyExp();
62  priors.Scale(1.0 / priors.Sum());
63 
64  am_nnet.SetPriors(priors);
65 
67  opts.max_nnet_batch_size = 20;
68  opts.acoustic_scale = 0.1;
69 
70  opts.pad_input = (rand() % 2 == 0);
71 
72  int32 num_input_frames = 400;
73  Matrix<BaseFloat> input_feats(num_input_frames, input_dim);
74  input_feats.SetRandn();
75 
76  OnlineMatrixFeature matrix_feature(input_feats);
77 
78  DecodableNnet2Online online_decodable(am_nnet, trans_model,
79  opts, &matrix_feature);
80 
81  DecodableAmNnet offline_decodable(trans_model, am_nnet,
82  CuMatrix<BaseFloat>(input_feats),
83  opts.pad_input,
84  opts.acoustic_scale);
85 
86  KALDI_ASSERT(online_decodable.NumFramesReady() ==
87  offline_decodable.NumFramesReady());
88  int32 num_frames = online_decodable.NumFramesReady(),
89  num_tids = trans_model.NumTransitionIds();
90 
91  for (int32 i = 0; i < 50; i++) {
92 
93  int32 t = rand() % num_frames, tid = 1 + rand() % num_tids;
94  BaseFloat l1 = online_decodable.LogLikelihood(t, tid),
95  l2 = offline_decodable.LogLikelihood(t, tid);
96  KALDI_ASSERT(ApproxEqual(l1, l2));
97  }
98 }
This class takes a Matrix and wraps it as an OnlineFeatureInterface: this can be useful wh...
HmmTopology GetDefaultTopology(const std::vector< int32 > &phones_in)
This function returns a HmmTopology object giving a normal 3-state topology, covering all phones in t...
ContextDependency * GenRandContextDependencyLarge(const std::vector< int32 > &phone_ids, int N, int P, bool ensure_all_covered, std::vector< int32 > *hmm_lengths)
GenRandContextDependencyLarge is like GenRandContextDependency but generates a larger tree with speci...
Definition: context-dep.cc:97
A class for storing topology information for phones.
Definition: hmm-topology.h:94
Nnet * GenRandomNnet(int32 input_dim, int32 output_dim)
This function generates a random neural net, for testing purposes.
Definition: nnet-nnet.cc:772
virtual int32 NumPdfs() const
NumPdfs() returns the number of acoustic pdfs (they are numbered 0.. NumPdfs()-1).
Definition: context-dep.h:71
float BaseFloat
Definition: kaldi-types.h:29
This Decodable object for class nnet2::AmNnet takes feature input from class OnlineFeatureInterface, unlike, say, class DecodableAmNnet which takes feature input from a matrix.
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
DecodableAmNnet is a decodable object that decodes with a neural net acoustic model of type AmNnet...
static bool ApproxEqual(float a, float b, float relative_tolerance=0.001)
return abs(a - b) <= relative_tolerance * (abs(a)+abs(b)).
Definition: kaldi-math.h:262
void kaldi::nnet2::UnitTestParsing ( )

Definition at line 696 of file nnet-component-test.cc.

References rnnlm::i, KALDI_ASSERT, and ParseFromString().

Referenced by main().

696  {
697  int32 i;
698  BaseFloat f;
699  bool b;
700  std::vector<int32> v;
701  std::string s = "x=y";
702  KALDI_ASSERT(ParseFromString("foo", &s, &i) == false
703  && s == "x=y");
704  KALDI_ASSERT(ParseFromString("foo", &s, &f) == false
705  && s == "x=y");
706  KALDI_ASSERT(ParseFromString("foo", &s, &v) == false
707  && s == "x=y");
708  KALDI_ASSERT(ParseFromString("foo", &s, &b) == false
709  && s == "x=y");
710  {
711  std::string s = "x=1";
712  KALDI_ASSERT(ParseFromString("x", &s, &i) == true
713  && i == 1 && s == "");
714  s = "a=b x=1";
715  KALDI_ASSERT(ParseFromString("x", &s, &i) == true
716  && i == 1 && s == "a=b");
717  }
718  {
719  std::string s = "foo=false";
720  KALDI_ASSERT(ParseFromString("foo", &s, &b) == true
721  && b == false && s == "");
722  s = "x=y foo=true a=b";
723  KALDI_ASSERT(ParseFromString("foo", &s, &b) == true
724  && b == true && s == "x=y a=b");
725  }
726 
727  {
728  std::string s = "foobar x=1";
729  KALDI_ASSERT(ParseFromString("x", &s, &f) == true
730  && f == 1.0 && s == "foobar");
731  s = "a=b x=1 bxy";
732  KALDI_ASSERT(ParseFromString("x", &s, &f) == true
733  && f == 1.0 && s == "a=b bxy");
734  }
735  {
736  std::string s = "x=1:2:3";
737  KALDI_ASSERT(ParseFromString("x", &s, &v) == true
738  && v.size() == 3 && v[0] == 1 && v[1] == 2 && v[2] == 3
739  && s == "");
740  s = "a=b x=1:2:3 c=d";
741  KALDI_ASSERT(ParseFromString("x", &s, &v) == true
742  && f == 1.0 && s == "a=b c=d");
743  }
744 
745 }
bool ParseFromString(const std::string &name, std::string *string, int32 *param)
Functions used in Init routines.
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void kaldi::nnet2::UnitTestPnormComponent ( )

Definition at line 286 of file nnet-component-test.cc.

References rnnlm::i, KALDI_ERR, KALDI_WARN, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

286  {
287  // We're testing that the gradients are computed correctly:
288  // the input gradients and the model gradients.
289 
290  int32 num_fail = 0, num_tries = 4;
291  for (int32 i = 0; i < num_tries; i++) {
292  try {
293  int32 output_dim = 10 + Rand() % 20,
294  group_size = 1 + Rand() % 10,
295  input_dim = output_dim * group_size;
296  BaseFloat p = 1.0 + 0.1 * (Rand() % 20);
297 
298  PnormComponent component(input_dim, output_dim, p);
300  } catch (...) {
301  KALDI_WARN << "Ignoring test failure in UnitTestPnormComponent().";
302  num_fail++;
303  }
304  }
305  if (num_fail >= num_tries/2) {
306  KALDI_ERR << "Too many test failures.";
307  }
308 }
float BaseFloat
Definition: kaldi-types.h:29
#define KALDI_ERR
Definition: kaldi-error.h:127
#define KALDI_WARN
Definition: kaldi-error.h:130
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestPreconditionDirections ( )

Definition at line 26 of file nnet-precondition-test.cc.

References CuSpMatrix< Real >::AddMat2(), CuVectorBase< Real >::AddSpVec(), CuSpMatrix< Real >::AddVec2(), CuVectorBase< Real >::ApproxEqual(), CuSpMatrix< Real >::Invert(), KALDI_ASSERT, kaldi::kTrans, rnnlm::n, PreconditionDirections(), kaldi::Rand(), CuPackedMatrix< Real >::ScaleDiag(), CuMatrixBase< Real >::SetRandn(), and CuPackedMatrix< Real >::SetUnit().

Referenced by main().

26  {
27  MatrixIndexT N = 2 + Rand() % 30,
28  D = 1 + Rand() % 20;
29  BaseFloat lambda = 0.1;
30  CuMatrix<BaseFloat> R(N, D), P(N, D);
31  R.SetRandn();
32  P.SetRandn(); // contents should be overwritten.
33 
34  PreconditionDirections(R, lambda, &P);
35  // The rest of this function will do the computation the function is doing in
36  // a different, less efficient way and compare with the function call.
37 
39  G.SetUnit();
40  G.ScaleDiag(lambda);
41  // G += R^T R.
42  G.AddMat2(1.0/(N-1), R, kTrans, 1.0);
43 
44  for (int32 n = 0; n < N; n++) {
47  Gn.AddVec2(-1.0/(N-1), rn); // subtract the
48  // outer product of "this" vector.
49  Gn.Invert();
51  CuVector<BaseFloat> pn_compare(D);
52  pn_compare.AddSpVec(1.0, Gn, rn, 0.0);
53  KALDI_ASSERT(pn.ApproxEqual(pn_compare, 0.1));
54  }
55 }
void PreconditionDirections(const CuMatrixBase< BaseFloat > &R, double lambda, CuMatrixBase< BaseFloat > *P)
See below for comment.
float BaseFloat
Definition: kaldi-types.h:29
int32 MatrixIndexT
Definition: matrix-common.h:96
struct rnnlm::@11::@12 n
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void kaldi::nnet2::UnitTestPreconditionDirectionsOnline ( )

Definition at line 261 of file nnet-precondition-online-test.cc.

References CuVectorBase< Real >::AddDiagMatMat(), MatrixBase< Real >::AddVecVec(), kaldi::AssertEqual(), KALDI_ASSERT, kaldi::kNoTrans, kaldi::kTrans, CuMatrixBase< Real >::NumRows(), OnlinePreconditionerSimple::PreconditionDirections(), OnlinePreconditioner::PreconditionDirections(), kaldi::Rand(), kaldi::RandInt(), VectorBase< Real >::Scale(), MatrixBase< Real >::Set(), VectorBase< Real >::SetRandn(), MatrixBase< Real >::SetRandn(), OnlinePreconditionerSimple::SetRank(), OnlinePreconditioner::SetRank(), kaldi::TraceMatMat(), and OnlinePreconditioner::TurnOnDebug().

Referenced by main().

261  {
262  MatrixIndexT R = 1 + Rand() % 30, // rank of correction
263  N = (2 * R) + Rand() % 30, // batch size
264  D = R + 1 + Rand() % 20; // problem dimension. Must be > R.
265 
266  // Test sometimes with features that are all-zero or all-one; this will
267  // help to make sure low-rank or zero input doesn't crash the code.
268  bool zero = false;
269  bool one = false;
270  if (Rand() % 3 == 0) zero = true;
271  //else if (Rand() % 2 == 0) one = true;
272 
273  CuVector<BaseFloat> row_prod1(N), row_prod2(N);
274  BaseFloat gamma1, gamma2;
275  BaseFloat big_eig_factor = RandInt(1, 20);
276  big_eig_factor = big_eig_factor * big_eig_factor;
277  Vector<BaseFloat> big_eig_vector(D);
278  big_eig_vector.SetRandn();
279  big_eig_vector.Scale(big_eig_factor);
280 
281  OnlinePreconditionerSimple preconditioner1;
282  OnlinePreconditioner preconditioner2;
283  preconditioner1.SetRank(R);
284  preconditioner2.SetRank(R);
285  preconditioner2.TurnOnDebug();
286 
287  int32 num_iters = 100;
288  for (int32 iter = 0; iter < num_iters; iter++) {
289  Matrix<BaseFloat> M_cpu(N, D);
290  if (one) M_cpu.Set(1.0);
291  else if (!zero) {
292  M_cpu.SetRandn();
293  Vector<BaseFloat> rand_vec(N);
294  rand_vec.SetRandn();
295  M_cpu.AddVecVec(1.0, rand_vec, big_eig_vector);
296  }
297  CuMatrix<BaseFloat> M(M_cpu);
298 
299  CuMatrix<BaseFloat> Mcopy1(M), Mcopy2(M);
300 
301  preconditioner1.PreconditionDirections(&Mcopy1, &row_prod1, &gamma1);
302 
303  preconditioner2.PreconditionDirections(&Mcopy2, &row_prod2, &gamma2);
304 
305  BaseFloat trace1 = TraceMatMat(M, M, kTrans),
306  trace2 = TraceMatMat(Mcopy1, Mcopy1, kTrans);
307  AssertEqual(trace1, trace2 * gamma2 * gamma2, 1.0e-02);
308 
309  AssertEqual(Mcopy1, Mcopy2);
310  AssertEqual<BaseFloat>(row_prod1, row_prod2, 1.0e-02);
311  AssertEqual(gamma1, gamma2, 1.0e-02);
312 
313  // make sure positive definite
314  CuVector<BaseFloat> inner_prods(M.NumRows());
315  inner_prods.AddDiagMatMat(1.0, M, kNoTrans, Mcopy1, kTrans, 0.0);
316  KALDI_ASSERT(inner_prods.Min() >= 0.0);
317  }
318  return;
319 }
void AddDiagMatMat(Real alpha, const CuMatrixBase< Real > &M, MatrixTransposeType transM, const CuMatrixBase< Real > &N, MatrixTransposeType transN, Real beta=1.0)
Add the diagonal of a matrix product: *this = diag(M N), assuming the "trans" arguments are both kNoT...
Definition: cu-vector.cc:544
void PreconditionDirections(CuMatrixBase< BaseFloat > *R, CuVectorBase< BaseFloat > *row_prod, BaseFloat *scale)
float BaseFloat
Definition: kaldi-types.h:29
int32 MatrixIndexT
Definition: matrix-common.h:96
Real TraceMatMat(const MatrixBase< Real > &A, const MatrixBase< Real > &B, MatrixTransposeType trans)
We need to declare this here as it will be a friend function.
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void PreconditionDirections(CuMatrixBase< BaseFloat > *R, CuVectorBase< BaseFloat > *row_prod, BaseFloat *scale)
static void AssertEqual(float a, float b, float relative_tolerance=0.001)
assert abs(a - b) <= relative_tolerance * (abs(a)+abs(b))
Definition: kaldi-math.h:273
Keywords for search: natural gradient, naturalgradient, NG-SGD.
int32 RandInt(int32 min_val, int32 max_val, struct RandomState *state)
Definition: kaldi-math.cc:94
void kaldi::nnet2::UnitTestScaleComponent ( )

Definition at line 461 of file nnet-component-test.cc.

References ScaleComponent::Init(), ScaleComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

461  {
462  int32 dim = 1 + Rand() % 10;
463  BaseFloat scale = 0.1 + Rand() % 3;
464  {
465  ScaleComponent component;
466  if (Rand() % 2 == 0) {
467  component.Init(dim, scale);
468  } else {
469  std::ostringstream str;
470  str << "dim=" << dim << " scale=" << scale;
471  component.InitFromString(str.str());
472  }
474  }
475 }
virtual void InitFromString(std::string args)
Initialize, typically from a line of a config file.
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void Init(int32 dim, BaseFloat scale)
void kaldi::nnet2::UnitTestSigmoidComponent ( )

Definition at line 227 of file nnet-component-test.cc.

References NonlinearComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

227  {
228  // We're testing that the gradients are computed correctly:
229  // the input gradients and the model gradients.
230 
231  int32 input_dim = 10 + Rand() % 50;
232  {
233  SigmoidComponent sigmoid_component(input_dim);
234  UnitTestGenericComponentInternal(sigmoid_component);
235  }
236  {
237  SigmoidComponent sigmoid_component;
238  sigmoid_component.InitFromString("dim=15");
239  UnitTestGenericComponentInternal(sigmoid_component);
240  }
241 }
virtual void InitFromString(std::string args)
We implement InitFromString at this level.
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
void UnitTestGenericComponentInternal(const Component &component)
void kaldi::nnet2::UnitTestSolvePackingProblem ( )

Definition at line 29 of file nnet-example-functions-test.cc.

References rnnlm::i, rnnlm::j, KALDI_ASSERT, kaldi::Rand(), SolvePackingProblem(), and kaldi::SortAndUniq().

Referenced by main().

29  {
30  size_t size = Rand() % 20;
31  std::vector<BaseFloat> item_costs;
32  for (size_t i = 0; i < size; i++) {
33  item_costs.push_back(0.5 * (Rand() % 15));
34  }
35  BaseFloat max_cost = 0.66 + Rand() % 5;
36 
37  std::vector<std::vector<size_t> > groups;
38  SolvePackingProblem(max_cost, item_costs, &groups);
39 
40  std::vector<size_t> all_indices;
41  for (size_t i = 0; i < groups.size(); i++) {
42  BaseFloat this_group_cost = 0.0;
43  for (size_t j = 0; j < groups[i].size(); j++) {
44  size_t index = groups[i][j];
45  all_indices.push_back(index);
46  this_group_cost += item_costs[index];
47  }
48  KALDI_ASSERT(!groups[i].empty());
49  KALDI_ASSERT(groups[i].size() == 1 || this_group_cost <= max_cost);
50  }
51  SortAndUniq(&all_indices);
52  KALDI_ASSERT(all_indices.size() == size);
53  if (!all_indices.empty())
54  KALDI_ASSERT(all_indices.back() + 1 == size);
55 }
void SortAndUniq(std::vector< T > *vec)
Sorts and uniq's (removes duplicates) from a vector.
Definition: stl-utils.h:39
float BaseFloat
Definition: kaldi-types.h:29
int Rand(struct RandomState *state)
Definition: kaldi-math.cc:44
#define KALDI_ASSERT(cond)
Definition: kaldi-error.h:169
void SolvePackingProblem(BaseFloat max_cost, const std::vector< BaseFloat > &costs, std::vector< std::vector< size_t > > *groups)
This function solves the "packing problem" using the "first fit" algorithm.
void kaldi::nnet2::UnitTestSpliceComponent ( )

Definition at line 747 of file nnet-component-test.cc.

References rnnlm::i, SpliceComponent::Init(), KALDI_LOG, kaldi::RandInt(), and UnitTestGenericComponentInternal().

Referenced by main().

747  {
748  int32 feat_dim = RandInt(1, 20),
749  const_dim = RandInt(0, 10),
7