Classes
class	AdditiveNoiseComponent
	This is a bit similar to dropout but adding (not multiplying) Gaussian noise with a given standard deviation. More...

class	AffineComponent

class	AffineComponentPreconditioned

class	AffineComponentPreconditionedOnline
	Keywords: natural gradient descent, NG-SGD, naturalgradient. More...

class	AmNnet

class	BlockAffineComponent

class	BlockAffineComponentPreconditioned

class	ChunkInfo
	ChunkInfo is a class whose purpose is to describe the structure of matrices holding features. More...

class	Component
	Abstract class, basic element of the network, it is a box with defined inputs, outputs, and tranformation functions interface. More...

class	Convolutional1dComponent
	Convolutional1dComponent implements convolution over frequency axis. More...

class	DctComponent
	Discrete cosine transform. More...

class	DecodableAmNnet
	DecodableAmNnet is a decodable object that decodes with a neural net acoustic model of type AmNnet. More...

class	DecodableAmNnetParallel
	This version of DecodableAmNnet is intended for a version of the decoder that processes different utterances with multiple threads. More...

class	DecodableNnet2Online
	This Decodable object for class nnet2::AmNnet takes feature input from class OnlineFeatureInterface, unlike, say, class DecodableAmNnet which takes feature input from a matrix. More...

struct	DecodableNnet2OnlineOptions

class	DiscriminativeExampleSplitter
	For each frame, judge: More...

class	DiscriminativeExamplesRepository
	This struct stores neural net training examples to be used in multi-threaded training. More...

struct	DiscriminativeNnetExample
	This struct is used to store the information we need for discriminative training (MMI or MPE). More...

class	DiscTrainParallelClass

class	DoBackpropParallelClass

class	DropoutComponent
	This Component, if present, randomly zeroes half of the inputs and multiplies the other half by two. More...

class	ExamplesRepository
	This class stores neural net training examples to be used in multi-threaded training. More...

class	FastNnetCombiner

class	FisherComputationClass

class	FixedAffineComponent
	FixedAffineComponent is an affine transform that is supplied at network initialization time and is not trainable. More...

class	FixedBiasComponent
	FixedBiasComponent applies a fixed per-element bias; it's similar to the AddShift component in the nnet1 setup (and only needed for nnet1 model conversion. More...

class	FixedLinearComponent
	FixedLinearComponent is a linear transform that is supplied at network initialization time and is not trainable. More...

class	FixedScaleComponent
	FixedScaleComponent applies a fixed per-element scale; it's similar to the Rescale component in the nnet1 setup (and only needed for nnet1 model conversion). More...

class	LimitRankClass

class	LogSoftmaxComponent

class	MaxoutComponent

class	MaxpoolingComponent
	MaxPoolingComponent : Maxpooling component was firstly used in ConvNet for selecting an representative activation in an area. More...

class	Nnet

struct	NnetCombineAconfig

struct	NnetCombineConfig
	Configuration class that controls neural net combination, where we combine a number of neural nets, trying to find for each layer the optimal weighted combination of the different neural-net parameters. More...

struct	NnetCombineFastConfig
	Configuration class that controls neural net combination, where we combine a number of neural nets, trying to find for each layer the optimal weighted combination of the different neural-net parameters. More...

class	NnetComputer

struct	NnetDiscriminativeStats

struct	NnetDiscriminativeUpdateOptions

class	NnetDiscriminativeUpdater

class	NnetEnsembleTrainer

struct	NnetEnsembleTrainerConfig

struct	NnetExample
	NnetExample is the input data and corresponding label (or labels) for one or more frames of input, used for standard cross-entropy training of neural nets (and possibly for other objective functions). More...

class	NnetExampleBackgroundReader

struct	NnetFixConfig

struct	NnetLimitRankOpts

struct	NnetMixupConfig

class	NnetOnlineComputer

struct	NnetRescaleConfig

class	NnetRescaler

struct	NnetShrinkConfig
	Configuration class that controls neural net "shrinkage" which is actually a scaling on the parameters of each of the updatable layers. More...

struct	NnetSimpleTrainerConfig

class	NnetStats

struct	NnetStatsConfig

class	NnetUpdater

struct	NnetWidenConfig
	Configuration class that controls neural net "widening", which means increasing the dimension of the hidden layers of an already-trained neural net. More...

class	NonlinearComponent
	This kind of Component is a base-class for things like sigmoid and softmax. More...

class	NormalizeComponent

class	OnlinePreconditioner
	Keywords for search: natural gradient, naturalgradient, NG-SGD. More...

class	OnlinePreconditionerSimple

class	PermuteComponent
	PermuteComponent does a permutation of the dimensions (by default, a fixed random permutation, but it may be specified). More...

class	PnormComponent

class	PowerComponent
	Take the absoute values of an input vector to a power. More...

class	RandomComponent

class	RectifiedLinearComponent

class	ScaleComponent

class	SigmoidComponent

class	SoftHingeComponent

class	SoftmaxComponent

class	SpliceComponent
	Splices a context window of frames together [over time]. More...

class	SpliceMaxComponent
	This is as SpliceComponent but outputs the max of any of the inputs (taking the max across time). More...

struct	SplitDiscriminativeExampleConfig
	Config structure for SplitExample, for splitting discriminative training examples. More...

struct	SplitExampleStats
	This struct exists only for diagnostic purposes. More...

class	SumGroupComponent

class	TanhComponent

class	UpdatableComponent
	Class UpdatableComponent is a Component which has trainable parameters and contains some global parameters for stochastic gradient descent (learning rate, L2 regularization constant). More...

Functions
void	UnitTestAmNnet ()

static void	GetUpdateDirection (const std::vector< Nnet > &nnets, Nnet *direction)

static void	AddDirection (const Nnet &orig_nnet, const Nnet &direction, const VectorBase< BaseFloat > &scales, Nnet *dest)
	Sets "dest" to orig_nnet plus "direction", with each updatable component of "direction" first scaled by the appropriate scale. More...

static BaseFloat	ComputeObjfAndGradient (const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const Nnet &orig_nnet, const Nnet &direction, Vector< double > *gradient)

void	CombineNnetsA (const NnetCombineAconfig &config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)

void	CombineNnetsFast (const NnetCombineFastConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets_in, Nnet *nnet_out)

static void	CombineNnets (const Vector< BaseFloat > &scale_params, const std::vector< Nnet > &nnets, Nnet *dest)

static int32	GetInitialModel (const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets)
	Returns an integer saying which model to use: either 0 ... More...

static void	GetInitialScaleParams (const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Vector< double > *scale_params)

static double	ComputeObjfAndGradient (const std::vector< NnetExample > &validation_set, const Vector< double > &scale_params, const std::vector< Nnet > &nnets, bool debug, Vector< double > *gradient)

void	CombineNnets (const NnetCombineConfig &combine_config, const std::vector< NnetExample > &validation_set, const std::vector< Nnet > &nnets, Nnet *nnet_out)

static void	GiveNnetCorrectTopology (Nnet nnet, AffineComponent affine_component, SoftmaxComponent softmax_component, SumGroupComponent *sum_group_component)
	This function makes sure the neural net ends with a SumGroupComponent. More...

void	MixupNnet (const NnetMixupConfig &mixup_config, Nnet *nnet)
	This function works as follows. More...

void	UnitTestGenericComponentInternal (const Component &component, const ChunkInfo in_info, const ChunkInfo out_info)

void	UnitTestGenericComponentInternal (const Component &component)

void	UnitTestSigmoidComponent ()

template<class T >
void	UnitTestGenericComponent (std::string extra_str="")

void	UnitTestMaxoutComponent ()

void	UnitTestPnormComponent ()

void	UnitTestMaxpoolingComponent ()

void	UnitTestAffineComponent ()

void	UnitTestConvolutional1dComponent ()

void	UnitTestDropoutComponent ()

void	UnitTestAdditiveNoiseComponent ()

void	UnitTestScaleComponent ()

void	UnitTestAffineComponentPreconditioned ()

void	UnitTestAffineComponentPreconditionedOnline ()

void	UnitTestBlockAffineComponent ()

void	UnitTestBlockAffineComponentPreconditioned ()

void	UnitTestSumGroupComponent ()

void	UnitTestDctComponent ()

void	UnitTestFixedLinearComponent ()

void	UnitTestFixedAffineComponent ()

void	UnitTestFixedScaleComponent ()

void	UnitTestFixedBiasComponent ()

void	UnitTestParsing ()

void	UnitTestSpliceComponent ()

void	BasicDebugTestForSpliceMax (bool output=false)

static void	ExpectOneOrTwoTokens (std::istream &is, bool binary, const std::string &token1, const std::string &token2)

bool	ParseFromString (const std::string &name, std::string string, int32 param)
	Functions used in Init routines. More...

bool	ParseFromString (const std::string &name, std::string string, bool param)
	This version is for parameters of type bool, which can appear as any string beginning with f, F, t or T. More...

bool	ParseFromString (const std::string &name, std::string string, BaseFloat param)
	This version is for parameters of type BaseFloat. More...

bool	ParseFromString (const std::string &name, std::string string, std::string param)

bool	ParseFromString (const std::string &name, std::string string, std::vector< int32 > param)
	This version is for parameters of type std::vector<int32>; it expects them as a colon-separated list, without spaces. More...

void	NnetDiscriminativeUpdateParallel (const AmNnet &am_nnet, const TransitionModel &tmodel, const NnetDiscriminativeUpdateOptions &opts, int32 num_threads, SequentialDiscriminativeNnetExampleReader example_reader, Nnet nnet_to_update, NnetDiscriminativeStats *stats)

void	NnetDiscriminativeUpdate (const AmNnet &am_nnet, const TransitionModel &tmodel, const NnetDiscriminativeUpdateOptions &opts, const DiscriminativeNnetExample &eg, Nnet nnet_to_update, NnetDiscriminativeStats stats)
	Does the neural net computation, lattice forward-backward, and backprop, for either the MMI, MPFE or SMBR objective functions. More...

void	UnitTestNnetCompute ()

void	UnitTestNnetComputeChunked ()

void	NnetComputation (const Nnet &nnet, const CuMatrixBase< BaseFloat > &input, bool pad_input, CuMatrixBase< BaseFloat > *output)
	Does the basic neural net computation, on a sequence of data (e.g. More...

void	NnetComputationChunked (const Nnet &nnet, const CuMatrixBase< BaseFloat > &input, int32 chunk_size, CuMatrixBase< BaseFloat > *output)
	Does the basic neural net computation, on a sequence of data (e.g. More...

BaseFloat	NnetGradientComputation (const Nnet &nnet, const CuMatrixBase< BaseFloat > &input, bool pad_input, const Posterior &pdf_post, Nnet *nnet_to_update)

BaseFloat	NnetGradientComputation (const Nnet &nnet, const MatrixBase< BaseFloat > &input, bool pad_input, BaseFloat utterance_weight, const std::vector< int32 > &labels, Nnet *nnet_to_update)
	Does the neural net computation and backprop, given input and labels. More...

void	UnitTestSolvePackingProblem ()

bool	LatticeToDiscriminativeExample (const std::vector< int32 > &alignment, const Matrix< BaseFloat > &feats, const CompactLattice &clat, BaseFloat weight, int32 left_context, int32 right_context, DiscriminativeNnetExample *eg)
	Converts lattice to discriminative training example. More...

void	SplitDiscriminativeExample (const SplitDiscriminativeExampleConfig &config, const TransitionModel &tmodel, const DiscriminativeNnetExample &eg, std::vector< DiscriminativeNnetExample > egs_out, SplitExampleStats stats_out)
	Split a "discriminative example" into multiple pieces, splitting where the lattice has "pinch points". More...

void	ExciseDiscriminativeExample (const SplitDiscriminativeExampleConfig &config, const TransitionModel &tmodel, const DiscriminativeNnetExample &eg, std::vector< DiscriminativeNnetExample > egs_out, SplitExampleStats stats_out)
	Remove unnecessary frames from discriminative training example. More...

void	UpdateHash (const TransitionModel &tmodel, const DiscriminativeNnetExample &eg, std::string criterion, bool drop_frames, bool one_silence_class, Matrix< double > hash, double num_weight, double den_weight, double tot_t)
	This function is used in code that tests the functionality that we provide here, about splitting and excising nnet examples. More...

void	ExampleToPdfPost (const TransitionModel &tmodel, const std::vector< int32 > &silence_phones, std::string criterion, bool drop_frames, bool one_silence_class, const DiscriminativeNnetExample &eg, Posterior *post)
	Given a discriminative training example, this function works out posteriors at the pdf level (note: these are "discriminative-training posteriors" that may be positive or negative. More...

void	SolvePackingProblem (BaseFloat max_cost, const std::vector< BaseFloat > &costs, std::vector< std::vector< size_t > > *groups)
	This function solves the "packing problem" using the "first fit" algorithm. More...

void	AppendDiscriminativeExamples (const std::vector< const DiscriminativeNnetExample * > &input, DiscriminativeNnetExample *output)
	Appends the given vector of examples (which must be non-empty) into a single output example (called by CombineExamples, which might be a more convenient interface). More...

void	CombineDiscriminativeExamples (int32 max_length, const std::vector< DiscriminativeNnetExample > &input, std::vector< DiscriminativeNnetExample > *output)
	This function is used to combine multiple discriminative-training examples (each corresponding to a segment of a lattice), into one. More...

bool	HasSimpleLabels (const NnetExample &eg, std::vector< int32 > *simple_labels)

void	FixNnet (const NnetFixConfig &config, Nnet *nnet)

int32	IndexOfSoftmaxLayer (const Nnet &nnet)
	If "nnet" has exactly one softmax layer, this function will return its index; otherwise it will return -1. More...

void	InsertComponents (const Nnet &src_nnet, int32 c, Nnet *dest_nnet)
	Inserts the components of one neural network into a particular place in the other one. More...

void	ReplaceLastComponents (const Nnet &src_nnet, int32 num_to_remove, Nnet *dest_nnet)
	Removes the last "num_to_remove" components and adds the components from "src_nnet". More...

void	LimitRankParallel (const NnetLimitRankOpts &opts, Nnet *nnet)
	This function limits the rank of each affine transform in the neural net, by zeroing out the smallest singular values. More...

void	UnitTestNnet ()

Nnet *	GenRandomNnet (int32 input_dim, int32 output_dim)
	This function generates a random neural net, for testing purposes. More...

void	UnitTestPreconditionDirectionsOnline ()

void	UnitTestPreconditionDirections ()

void	PreconditionDirections (const CuMatrixBase< BaseFloat > &R, double lambda, CuMatrixBase< BaseFloat > *P)
	See below for comment. More...

void	PreconditionDirectionsAlpha (const CuMatrixBase< BaseFloat > &R, double alpha, CuMatrixBase< BaseFloat > *P)
	This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections. More...

void	PreconditionDirectionsAlphaRescaled (const CuMatrixBase< BaseFloat > &R, double alpha, CuMatrixBase< BaseFloat > *P)
	This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections. More...

void	GetNnetStats (const NnetStatsConfig &config, const Nnet &nnet, std::vector< NnetStats > *stats)

double	DoBackpropParallel (const Nnet &nnet, int32 minibatch_size, SequentialNnetExampleReader example_reader, double tot_weight, Nnet *nnet_to_update)
	This function is similar to "DoBackprop" in nnet-update.h This function computes the objective function and either updates the model or computes parameter gradients. More...

double	DoBackpropSingleThreaded (const Nnet &nnet, int32 minibatch_size, const std::vector< NnetExample > &egs, double tot_weight, Nnet nnet_to_update)

double	DoBackpropParallel (const Nnet &nnet, int32 minibatch_size, int32 num_threads, const std::vector< NnetExample > &examples, double num_frames, Nnet nnet_to_update)
	This version of DoBackpropParallel takes a vector of examples, and will typically be used to compute the exact gradient. More...

double	ComputeNnetObjfParallel (const Nnet &nnet, int32 minibatch_size, int32 num_threads, const std::vector< NnetExample > &examples, double *num_frames)
	This is basically to clarify the fact that DoBackpropParallel will also work with nnet_to_update == NULL, and will compute the objf. More...

void	FormatNnetInput (const Nnet &nnet, const std::vector< NnetExample > &data, Matrix< BaseFloat > *mat)
	Takes the input to the nnet for a minibatch of examples, and formats as a single matrix. More...

BaseFloat	TotalNnetTrainingWeight (const std::vector< NnetExample > &egs)
	Returns the total weight summed over all the examples... More...

double	ComputeNnetObjf (const Nnet &nnet, const std::vector< NnetExample > &examples, double *tot_accuracy=NULL)
	Computes objective function over a minibatch. More...

double	DoBackprop (const Nnet &nnet, const std::vector< NnetExample > &examples, Nnet nnet_to_update, double tot_accuracy=NULL)
	This function computes the objective function and either updates the model or adds to parameter gradients. More...

double	DoBackprop (const Nnet &nnet, const std::vector< NnetExample > &examples, Matrix< BaseFloat > examples_formatted, Nnet nnet_to_update, double *tot_accuracy=NULL)
	This version of DoBackprop allows you to separately call FormatNnetInput and provide the result to DoBackprop; this can be useful when using GPUs because the call to FormatNnetInput can be in a separate thread from the one that uses the GPU. More...

double	ComputeNnetGradient (const Nnet &nnet, const std::vector< NnetExample > &examples, int32 batch_size, Nnet *gradient)
	ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example into batches and calls DoBackprop() on each. More...

double	ComputeNnetObjf (const Nnet &nnet, const std::vector< NnetExample > &examples, int32 minibatch_size, double *tot_accuracy=NULL)
	This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computation. More...

void	UnitTestNnetDecodable ()

void	RescaleNnet (const NnetRescaleConfig &rescale_config, const std::vector< NnetExample > &examples, Nnet *nnet)

static BaseFloat	ComputeObjfAndGradient (const std::vector< NnetExample > &validation_set, const Vector< double > &log_scale_params, const Nnet &nnet, Vector< double > *gradient)

void	ShrinkNnet (const NnetShrinkConfig &shrink_config, const std::vector< NnetExample > &validation_set, Nnet *nnet)

static Int32Pair	MakePair (int32 first, int32 second)

int64	TrainNnetSimple (const NnetSimpleTrainerConfig &config, Nnet nnet, SequentialNnetExampleReader reader, double tot_weight=NULL, double tot_logprob=NULL)
	Train on all the examples it can read from the reader. More...

void	WidenNnet (const NnetWidenConfig &widen_config, Nnet *nnet)
	This function widens a neural network by increasing the hidden-layer dimensions to the target. More...

BaseFloat	KlDivergence (const Vector< BaseFloat > &p, const Vector< BaseFloat > &q)

void	PrintPriorDiagnostics (const Vector< BaseFloat > &old_priors, const Vector< BaseFloat > &new_priors)

int32	GetCount (double expected_count)

void	AverageConstPart (int32 const_feat_dim, DiscriminativeNnetExample *eg)

static void	ProcessFile (const MatrixBase< BaseFloat > &feats, const Posterior &pdf_post, const std::string &utt_id, int32 left_context, int32 right_context, int32 num_frames, int32 const_feat_dim, int64 num_frames_written, int64 num_egs_written, NnetExampleWriter *example_writer)

static void	ProcessFile (const MatrixBase< BaseFloat > &feats, const Posterior &pdf_post, const std::string &utt_id, const Vector< BaseFloat > &weights, int32 left_context, int32 right_context, int32 const_feat_dim, BaseFloat keep_proportion, BaseFloat weight_threshold, bool use_frame_selection, bool use_frame_weights, int64 num_frames_written, int64 num_frames_skipped, NnetExampleWriter *example_writer)

void	SetMaxChange (BaseFloat max_change, Nnet *nnet)

void	SetPriors (const TransitionModel &tmodel, const Vector< double > &transition_accs, double prior_floor, AmNnet *am_nnet)

Variables
static bool	nnet_example_warned_left = false

static bool	nnet_example_warned_right = false

Typedef Documentation

◆ DiscriminativeNnetExampleWriter

typedef TableWriter<KaldiObjectHolder<DiscriminativeNnetExample > > DiscriminativeNnetExampleWriter

Definition at line 181 of file nnet-example.h.

◆ NnetExampleWriter

typedef TableWriter<KaldiObjectHolder<NnetExample > > NnetExampleWriter

Definition at line 92 of file nnet-example.h.

◆ RandomAccessDiscriminativeNnetExampleReader

typedef RandomAccessTableReader<KaldiObjectHolder<DiscriminativeNnetExample > > RandomAccessDiscriminativeNnetExampleReader

Definition at line 185 of file nnet-example.h.

◆ RandomAccessNnetExampleReader

typedef RandomAccessTableReader<KaldiObjectHolder<NnetExample > > RandomAccessNnetExampleReader

Definition at line 94 of file nnet-example.h.

◆ SequentialDiscriminativeNnetExampleReader

typedef SequentialTableReader<KaldiObjectHolder<DiscriminativeNnetExample > > SequentialDiscriminativeNnetExampleReader

Definition at line 183 of file nnet-example.h.

◆ SequentialNnetExampleReader

typedef SequentialTableReader<KaldiObjectHolder<NnetExample > > SequentialNnetExampleReader

Definition at line 93 of file nnet-example.h.

Function Documentation

◆ AddDirection()

static void kaldi::nnet2::AddDirection	(	const Nnet &	orig_nnet,
		const Nnet &	direction,
		const VectorBase< BaseFloat > &	scales,
		Nnet *	dest
	)

static

Sets "dest" to orig_nnet plus "direction", with each updatable component of "direction" first scaled by the appropriate scale.

Definition at line 52 of file combine-nnet-a.cc.

References Nnet::AddNnet().

Referenced by CombineNnetsA(), and ComputeObjfAndGradient().

                                      {
   *dest = orig_nnet;
   dest->AddNnet(scales, direction);
 }

◆ AppendDiscriminativeExamples()

void AppendDiscriminativeExamples	(	const std::vector< const DiscriminativeNnetExample * > &	input,
		DiscriminativeNnetExample *	output
	)

Appends the given vector of examples (which must be non-empty) into a single output example (called by CombineExamples, which might be a more convenient interface).

When combining examples it directly appends the features, and then adds a "fake" segment to the lattice and alignment in between, padding with transition-ids that are all ones. This is necessary in case the network needs acoustic context, and only because of a kind of limitation in the nnet training code that doesn't support varying 'chunk' sizes within a minibatch.

Will fail if all the input examples don't have the same weight (this will normally be 1.0 anyway), or if the feature dimension (i.e. basic feature dimension plus spk_info dimension) differs between the examples.

Definition at line 891 of file nnet-example-functions.cc.

References DiscriminativeNnetExample::den_lat, rnnlm::i, DiscriminativeNnetExample::input_frames, KALDI_ASSERT, kaldi::kUndefined, DiscriminativeNnetExample::left_context, DiscriminativeNnetExample::num_ali, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), CompactLatticeWeightTpl< WeightType, IntType >::One(), MatrixBase< Real >::Range(), Matrix< Real >::Resize(), DiscriminativeNnetExample::spk_info, and DiscriminativeNnetExample::weight.

Referenced by CombineDiscriminativeExamples(), and SplitExampleStats::SplitExampleStats().

                                        {
   KALDI_ASSERT(!input.empty());
   const DiscriminativeNnetExample &eg0 = *(input[0]);
   
   int32 dim = eg0.input_frames.NumCols() + eg0.spk_info.Dim(),
       left_context = eg0.left_context,
       num_frames = eg0.num_ali.size(),
       right_context = eg0.input_frames.NumRows() - num_frames - left_context;
 
   int32 tot_frames = eg0.input_frames.NumRows();  // total frames (appended,
                                                   // with context)
   for (size_t i = 1; i < input.size(); i++)
     tot_frames += input[i]->input_frames.NumRows();
 
   int32 arbitrary_tid = 1;  // arbitrary transition-id that we use to pad the
                             // num_ali and den_lat members between segments
                             // (since they're both the same, and the den-lat in
                             // those parts is linear, they contribute no
                             // derivative to the training).
   
   output->den_lat = eg0.den_lat;
   output->num_ali = eg0.num_ali;
   output->input_frames.Resize(tot_frames, dim, kUndefined);
   output->input_frames.Range(0, eg0.input_frames.NumRows(),
                              0, eg0.input_frames.NumCols()).CopyFromMat(eg0.input_frames);
   if (eg0.spk_info.Dim() != 0) {
     output->input_frames.Range(0, eg0.input_frames.NumRows(),
                                eg0.input_frames.NumCols(), eg0.spk_info.Dim()).
         CopyRowsFromVec(eg0.spk_info);
   }
   
   output->num_ali.reserve(tot_frames - left_context - right_context);
   output->weight = eg0.weight;
   output->left_context = eg0.left_context;
   output->spk_info.Resize(0);
 
   CompactLattice inter_segment_clat;
   int32 initial = inter_segment_clat.AddState(); // state 0.
   inter_segment_clat.SetStart(initial);
   
   std::vector<int32> inter_segment_ali(left_context + right_context);
   std::fill(inter_segment_ali.begin(), inter_segment_ali.end(), arbitrary_tid);
 
   CompactLatticeWeight final_weight = CompactLatticeWeight::One();
   final_weight.SetString(inter_segment_ali);
   inter_segment_clat.SetFinal(initial, final_weight);
   
   int32 feat_offset = eg0.input_frames.NumRows();
   
   for (size_t i = 1; i < input.size(); i++) {
     const DiscriminativeNnetExample &eg_i = *(input[i]);
         
     output->input_frames.Range(feat_offset, eg_i.input_frames.NumRows(),
                                0, eg_i.input_frames.NumCols()).CopyFromMat(
                                    eg_i.input_frames);
     if (eg_i.spk_info.Dim() != 0) {
       output->input_frames.Range(feat_offset, eg_i.input_frames.NumRows(),
                                  eg_i.input_frames.NumCols(),
                                  eg_i.spk_info.Dim()).CopyRowsFromVec(
                                      eg_i.spk_info);
       KALDI_ASSERT(eg_i.input_frames.NumCols() +
                    eg_i.spk_info.Dim() == dim);
     }
     
     output->num_ali.insert(output->num_ali.end(),
                            inter_segment_ali.begin(), inter_segment_ali.end());
     output->num_ali.insert(output->num_ali.end(),
                            eg_i.num_ali.begin(), eg_i.num_ali.end());
     Concat(&(output->den_lat), inter_segment_clat);
     Concat(&(output->den_lat), eg_i.den_lat);
     KALDI_ASSERT(output->weight == eg_i.weight);
     KALDI_ASSERT(output->left_context == eg_i.left_context);
     feat_offset += eg_i.input_frames.NumRows();
   }
   KALDI_ASSERT(feat_offset == tot_frames);
 }

◆ AverageConstPart()

void kaldi::nnet2::AverageConstPart	(	int32	const_feat_dim,
		DiscriminativeNnetExample *	eg
	)

Definition at line 42 of file nnet-copy-egs-discriminative.cc.

References DiscriminativeNnetExample::input_frames, KALDI_ASSERT, MatrixBase< Real >::NumCols(), MatrixBase< Real >::Range(), and DiscriminativeNnetExample::spk_info.

Referenced by main().

                                                      {
   if (eg->spk_info.Dim() != 0) {  // already has const part.
     KALDI_ASSERT(eg->spk_info.Dim() == const_feat_dim);
     // and nothing to do.
   } else {
     int32 dim = eg->input_frames.NumCols(),
         basic_dim = dim - const_feat_dim;
     KALDI_ASSERT(const_feat_dim < eg->input_frames.NumCols());
     Matrix<BaseFloat> mat(eg->input_frames);  // copy to non-compressed matrix.
     eg->input_frames = mat.Range(0, mat.NumRows(), 0, basic_dim);
     eg->spk_info.Resize(const_feat_dim);
     eg->spk_info.AddRowSumMat(1.0 / mat.NumRows(),
                               mat.Range(0, mat.NumRows(),
                                         basic_dim, const_feat_dim),
                               0.0);
   }
 }

◆ BasicDebugTestForSpliceMax()

void kaldi::nnet2::BasicDebugTestForSpliceMax ( bool output = false )

Definition at line 805 of file nnet-component-test.cc.

References SpliceMaxComponent::Backprop(), rnnlm::i, SpliceMaxComponent::Init(), KALDI_LOG, SpliceMaxComponent::OutputDim(), and SpliceMaxComponent::Propagate().

Referenced by main().

                                                    {
   int32 C=5,
         context_len=2,
         R= 3 + 2*context_len;
 
   SpliceMaxComponent *c = new SpliceMaxComponent();
   std::vector<int32> context(2 * context_len + 1);
   for (int32 i = -1 * context_len; i <= context_len; i++)
     context[i + context_len] = i;
   c->Init(C, context);
   CuMatrix<BaseFloat> in(R, C), in_deriv(R, C);
   CuMatrix<BaseFloat> out(R, c->OutputDim());
   ChunkInfo in_info = ChunkInfo(C, 1, 0, R - 1),
             out_info = ChunkInfo(C, 1, context_len, R - 1 - context_len);
 
   in.SetRandn();
   if (output)
     KALDI_LOG << in;
 
   c->Propagate(in_info, out_info, in, &out);
 
   if (output)
     KALDI_LOG << out;
 
   out.Set(5.0);
 
   if (output)
     KALDI_LOG << out;
 
   c->Backprop(in_info, out_info, in, in, out, c, &in_deriv);
 
   if (output)
     KALDI_LOG << in_deriv;
 
   delete c;
 }

◆ CombineDiscriminativeExamples()

void CombineDiscriminativeExamples	(	int32	max_length,
		const std::vector< DiscriminativeNnetExample > &	input,
		std::vector< DiscriminativeNnetExample > *	output
	)

This function is used to combine multiple discriminative-training examples (each corresponding to a segment of a lattice), into one.

It combines examples into groups such that each group will have a total length (number of rows of the feature matrix) less than or equal to max_length. However, if individual examples are longer than max_length they will still be processed; they will be given their own group.

See also the documentation for AppendDiscriminativeExamples() which gives more details on how we append the examples.

Will fail if all the input examples don't have the same weight (this will normally be 1.0 anyway).

If the spk_info variables are non-empty, it will move them into the features of the output, so the spk_info of the output will be empty but the appropriate speaker vectors will be appended to each row of the features.

Definition at line 970 of file nnet-example-functions.cc.

References AppendDiscriminativeExamples(), rnnlm::i, rnnlm::j, and SolvePackingProblem().

Referenced by main(), and SplitExampleStats::SplitExampleStats().

                                                   {
   
   std::vector<BaseFloat> costs(input.size());
   for (size_t i = 0; i < input.size(); i++)
     costs[i] = static_cast<BaseFloat>(input[i].input_frames.NumRows());
   std::vector<std::vector<size_t> > groups;
   SolvePackingProblem(max_length,
                       costs,
                       &groups);
   output->clear();
   output->resize(groups.size());
   for (size_t i = 0; i < groups.size(); i++) {
     std::vector<const DiscriminativeNnetExample*> group_egs;
     for (size_t j = 0; j < groups[i].size(); j++) {
       size_t index = groups[i][j];
       group_egs.push_back(&(input[index]));
     }
     AppendDiscriminativeExamples(group_egs, &((*output)[i]));
   }
 }

◆ CombineNnets() [1/2]

static void kaldi::nnet2::CombineNnets	(	const Vector< BaseFloat > &	scale_params,
		const std::vector< Nnet > &	nnets,
		Nnet *	dest
	)

static

Definition at line 28 of file combine-nnet.cc.

References Nnet::AddNnet(), KALDI_ASSERT, rnnlm::n, kaldi::nnet3::NumUpdatableComponents(), and Nnet::ScaleComponents().

Referenced by CombineNnets(), FastNnetCombiner::ComputeCurrentNnet(), ComputeObjfAndGradient(), FastNnetCombiner::FastNnetCombiner(), GetInitialModel(), FastNnetCombiner::GetInitialModel(), main(), and NnetCombineConfig::Register().

                                      {
   int32 num_nnets = nnets.size();
   KALDI_ASSERT(num_nnets >= 1);
   int32 num_uc = nnets[0].NumUpdatableComponents();
   KALDI_ASSERT(nnets[0].NumUpdatableComponents() >= 1);
 
 
   *dest = nnets[0];
   SubVector<BaseFloat> scale_params0(scale_params, 0, num_uc);
   dest->ScaleComponents(scale_params0);
   for (int32 n = 1; n < num_nnets; n++) {
     SubVector<BaseFloat> scale_params_n(scale_params, n * num_uc, num_uc);
     dest->AddNnet(scale_params_n, nnets[n]);
   }
 }

◆ CombineNnets() [2/2]

void CombineNnets	(	const NnetCombineConfig &	combine_config,
		const std::vector< NnetExample > &	validation_set,
		const std::vector< Nnet > &	nnets,
		Nnet *	nnet_out
	)

Definition at line 193 of file combine-nnet.cc.

References CombineNnets(), ComputeObjfAndGradient(), VectorBase< Real >::CopyFromVec(), MatrixBase< Real >::CopyRowsFromVec(), VectorBase< Real >::Dim(), OptimizeLbfgs< Real >::DoStep(), LbfgsOptions::first_step_impr, GetInitialScaleParams(), OptimizeLbfgs< Real >::GetProposedValue(), OptimizeLbfgs< Real >::GetValue(), rnnlm::i, NnetCombineConfig::initial_impr, KALDI_ASSERT, KALDI_LOG, KALDI_VLOG, LbfgsOptions::m, LbfgsOptions::minimize, NnetCombineConfig::num_bfgs_iters, and NnetCombineConfig::test_gradient.

                                   {
 
   Vector<double> scale_params;
 
   GetInitialScaleParams(combine_config,
                         validation_set,
                         nnets,
                         &scale_params);
 
   int32 dim = scale_params.Dim();
   KALDI_ASSERT(dim > 0);
   Vector<double> gradient(dim);
 
   double objf, initial_objf;
 
   LbfgsOptions lbfgs_options;
   lbfgs_options.minimize = false; // We're maximizing.
   lbfgs_options.m = dim; // Store the same number of vectors as the dimension
   // itself, so this is BFGS.
   lbfgs_options.first_step_impr = combine_config.initial_impr;
 
   OptimizeLbfgs<double> lbfgs(scale_params,
                               lbfgs_options);
 
   for (int32 i = 0; i < combine_config.num_bfgs_iters; i++) {
     scale_params.CopyFromVec(lbfgs.GetProposedValue());
     objf = ComputeObjfAndGradient(validation_set,
                                   scale_params,
                                   nnets,
                                   combine_config.test_gradient,
                                   &gradient);
 
     KALDI_VLOG(2) << "Iteration " << i << " scale-params = " << scale_params
                   << ", objf = " << objf << ", gradient = " << gradient;
 
     if (i == 0) initial_objf = objf;
 
     lbfgs.DoStep(objf, gradient);
   }
 
   scale_params.CopyFromVec(lbfgs.GetValue(&objf));
 
   Vector<BaseFloat> scale_params_float(scale_params);
 
   KALDI_LOG << "Combining nnets, validation objf per frame changed from "
             << initial_objf << " to " << objf;
 
   Matrix<BaseFloat> scale_params_mat(nnets.size(),
                                      nnets[0].NumUpdatableComponents());
   scale_params_mat.CopyRowsFromVec(scale_params_float);
   KALDI_LOG << "Final scale factors are " << scale_params_mat;
 
   CombineNnets(scale_params_float, nnets, nnet_out);
 }

◆ CombineNnetsA()

void CombineNnetsA	(	const NnetCombineAconfig &	config,
		const std::vector< NnetExample > &	validation_set,
		const std::vector< Nnet > &	nnets,
		Nnet *	nnet_out
	)

Definition at line 102 of file combine-nnet-a.cc.

References AddDirection(), ComputeObjfAndGradient(), VectorBase< Real >::CopyFromVec(), VectorBase< Real >::Dim(), OptimizeLbfgs< Real >::DoStep(), LbfgsOptions::first_step_length, Nnet::GetComponent(), OptimizeLbfgs< Real >::GetProposedValue(), GetUpdateDirection(), OptimizeLbfgs< Real >::GetValue(), rnnlm::i, NnetCombineAconfig::initial_step, rnnlm::j, KALDI_ASSERT, KALDI_LOG, KALDI_VLOG, KALDI_WARN, UpdatableComponent::LearningRate(), LbfgsOptions::m, NnetCombineAconfig::max_learning_rate_factor, NnetCombineAconfig::min_learning_rate, NnetCombineAconfig::min_learning_rate_factor, LbfgsOptions::minimize, NnetCombineAconfig::num_bfgs_iters, Nnet::NumComponents(), kaldi::nnet3::NumUpdatableComponents(), NnetCombineAconfig::overshoot, VectorBase< Real >::Set(), UpdatableComponent::SetLearningRate(), and NnetCombineAconfig::valid_impr_thresh.

Referenced by NnetCombineAconfig::Register().

                                    {
 
   Nnet direction; // the update direction = avg(nnets[1 ... N]) - nnets[0].
   GetUpdateDirection(nnets, &direction);
   
   Vector<double> scale_params(nnets[0].NumUpdatableComponents()); // initial
   // scale on "direction".
 
   int32 dim = scale_params.Dim();
   KALDI_ASSERT(dim > 0);
   Vector<double> gradient(dim);
   
   double objf, initial_objf, zero_objf;
 
   // Compute objf at zero; we don't actually need this gradient.
   zero_objf = ComputeObjfAndGradient(validation_set,
                                      scale_params,
                                      nnets[0],
                                      direction,
                                      &gradient);
   KALDI_LOG << "Objective function at old parameters is "
             << zero_objf;
   
   scale_params.Set(1.0); // start optimization from the average of the parameters.
 
   LbfgsOptions lbfgs_options;
   lbfgs_options.minimize = false; // We're maximizing.
   lbfgs_options.m = dim; // Store the same number of vectors as the dimension
   // itself, so this is BFGS.
   lbfgs_options.first_step_length = config.initial_step;
   
   OptimizeLbfgs<double> lbfgs(scale_params,
                               lbfgs_options);
   
   for (int32 i = 0; i < config.num_bfgs_iters; i++) {    
     scale_params.CopyFromVec(lbfgs.GetProposedValue());
     objf = ComputeObjfAndGradient(validation_set,
                                   scale_params,
                                   nnets[0],
                                   direction,
                                   &gradient);
 
     KALDI_VLOG(2) << "Iteration " << i << " scale-params = " << scale_params
                   << ", objf = " << objf << ", gradient = " << gradient;
     
     if (i == 0) initial_objf = objf;    
     lbfgs.DoStep(objf, gradient);
   }
 
   scale_params.CopyFromVec(lbfgs.GetValue(&objf));
 
   KALDI_LOG << "Combining nnets, after BFGS, validation objf per frame changed from "
             << zero_objf << " (no change), or " << initial_objf << " (default change), "
             << " to " << objf << "; scale factors on update direction are "
             << scale_params;
 
   BaseFloat objf_change = objf - zero_objf;
   KALDI_ASSERT(objf_change >= 0.0); // This is guaranteed by the L-BFGS code.
 
   if (objf_change < config.valid_impr_thresh) {
     // We'll overshoot.  To have a smooth transition between the two regimes, if
     // objf_change is close to valid_impr_thresh we don't overshoot as far.
     BaseFloat overshoot = config.overshoot,
         overshoot_max = config.valid_impr_thresh / objf_change; // >= 1.0.
     if (overshoot_max < overshoot) {
       KALDI_LOG << "Limiting overshoot from " << overshoot << " to " << overshoot_max
                 << " since the objf-impr " << objf_change << " is close to "
                 << "--valid-impr-thresh=" << config.valid_impr_thresh;
       overshoot = overshoot_max;
     }
     KALDI_ASSERT(overshoot < 2.0 && "--valid-impr-thresh must be < 2.0 or "
                  "it will lead to instability.");
     scale_params.Scale(overshoot);
 
     BaseFloat optimized_objf = objf;
     objf = ComputeObjfAndGradient(validation_set,
                                   scale_params,
                                   nnets[0],
                                   direction,
                                   &gradient);
 
     KALDI_LOG << "Combining nnets, after overshooting, validation objf changed "
               << "to " << objf << ".  Note: (zero, start, optimized) objfs were "
               << zero_objf << ", " << initial_objf << ", " << optimized_objf;
     if (objf < zero_objf) {
       // Note: this should not happen according to a quadratic approximation, and we
       // expect this branch to be taken only rarely if at all.
       KALDI_WARN << "After overshooting, objf was worse than not updating; not doing the "
                  << "overshoot. ";
      scale_params.Scale(1.0 / overshoot);
     }
   } // Else don't do the "overshoot" stuff.
   
   Vector<BaseFloat> scale_params_float(scale_params);
   // Output to "nnet_out":
   AddDirection(nnets[0], direction, scale_params_float, nnet_out);
 
   // Now update the neural net learning rates.
   int32 i = 0;
   for (int32 j = 0; j < nnet_out->NumComponents(); j++) {
     UpdatableComponent *uc =
         dynamic_cast<UpdatableComponent*>(&(nnet_out->GetComponent(j)));
     if (uc != NULL) {
       BaseFloat step_length = scale_params(i), factor = step_length;
       // Our basic rule is to update the learning rate by multiplying it
       // by "step_lenght", but this is subject to certain limits.
       if (factor < config.min_learning_rate_factor)
         factor = config.min_learning_rate_factor;
       if (factor > config.max_learning_rate_factor)
         factor = config.max_learning_rate_factor;
       BaseFloat new_learning_rate = factor * uc->LearningRate();
       if (new_learning_rate < config.min_learning_rate)
         new_learning_rate = config.min_learning_rate;
       KALDI_LOG << "For component " << j << ", step length was " << step_length
                 << ", updating learning rate by factor " << factor << ", changing "
                 << "learning rate from " << uc->LearningRate() << " to "
                 << new_learning_rate;
       uc->SetLearningRate(new_learning_rate);
       i++;
     }
   }
 }

◆ CombineNnetsFast()

void CombineNnetsFast	(	const NnetCombineFastConfig &	combine_config,
		const std::vector< NnetExample > &	validation_set,
		const std::vector< Nnet > &	nnets_in,
		Nnet *	nnet_out
	)

Definition at line 430 of file combine-nnet-fast.cc.

Referenced by main(), and NnetCombineFastConfig::Register().

                                       {
   // Everything happens in the initializer.
   FastNnetCombiner combiner(combine_config,
                             validation_set,
                             nnets_in,
                             nnet_out);
 }

◆ ComputeNnetGradient()

double ComputeNnetGradient	(	const Nnet &	nnet,
		const std::vector< NnetExample > &	examples,
		int32	batch_size,
		Nnet *	gradient
	)

ComputeNnetGradient is mostly used to compute gradients on validation sets; it divides the example into batches and calls DoBackprop() on each.

It returns the *average* objective function per frame.

Definition at line 302 of file nnet-update.cc.

References DoBackprop(), rnnlm::i, and Nnet::SetZero().

Referenced by ComputeObjfAndGradient(), and main().

                     {
   bool treat_as_gradient = true;
   gradient->SetZero(treat_as_gradient);
   std::vector<NnetExample> batch;
   batch.reserve(batch_size);
   double tot_objf = 0.0;
   for (int32 start_pos = 0;
        start_pos < static_cast<int32>(validation_set.size());
        start_pos += batch_size) {
     batch.clear();
     for (int32 i = start_pos;
          i < std::min(start_pos + batch_size,
                       static_cast<int32>(validation_set.size()));
          i++) {
       batch.push_back(validation_set[i]);
     }
     tot_objf += DoBackprop(nnet,
                            batch,
                            gradient);
   }
   return tot_objf / validation_set.size();
 }

◆ ComputeNnetObjf() [1/2]

double ComputeNnetObjf	(	const Nnet &	nnet,
		const std::vector< NnetExample > &	examples,
		double *	tot_accuracy = `NULL`
	)

Computes objective function over a minibatch.

Returns the *total* weighted objective function over the minibatch. If tot_accuracy != NULL, it outputs to that pointer the total (weighted) accuracy.

Definition at line 258 of file nnet-update.cc.

References NnetUpdater::ComputeForMinibatch().

Referenced by ComputeNnetObjf(), DoBackprop(), GetInitialModel(), main(), and DoBackpropParallelClass::operator()().

                                              {
   NnetUpdater updater(nnet, NULL);
   return updater.ComputeForMinibatch(examples, tot_accuracy);
 }

◆ ComputeNnetObjf() [2/2]

double ComputeNnetObjf	(	const Nnet &	nnet,
		const std::vector< NnetExample > &	examples,
		int32	minibatch_size,
		double *	tot_accuracy = `NULL`
	)

This version of ComputeNnetObjf breaks up the examples into multiple minibatches to do the computation.

Returns the *total* (weighted) objective function. If tot_accuracy != NULL, it outputs to that pointer the total (weighted) accuracy.

Definition at line 329 of file nnet-update.cc.

References ComputeNnetObjf(), and rnnlm::i.

                           {
   double tot_accuracy_tmp;
   if (tot_accuracy)
     *tot_accuracy = 0.0;
   std::vector<NnetExample> batch;
   batch.reserve(batch_size);
   double tot_objf = 0.0;
   for (int32 start_pos = 0;
        start_pos < static_cast<int32>(validation_set.size());
        start_pos += batch_size) {
     batch.clear();
     for (int32 i = start_pos;
          i < std::min(start_pos + batch_size,
                       static_cast<int32>(validation_set.size()));
          i++) {
       batch.push_back(validation_set[i]);
     }
     tot_objf += ComputeNnetObjf(nnet, batch,
                                 tot_accuracy != NULL ? &tot_accuracy_tmp : NULL);
     if (tot_accuracy)
       *tot_accuracy += tot_accuracy_tmp;
   }
   return tot_objf;
 }

◆ ComputeNnetObjfParallel()

double kaldi::nnet2::ComputeNnetObjfParallel	(	const Nnet &	nnet,
		int32	minibatch_size,
		int32	num_threads,
		const std::vector< NnetExample > &	examples,
		double *	num_frames
	)

inline

This is basically to clarify the fact that DoBackpropParallel will also work with nnet_to_update == NULL, and will compute the objf.

Both versions of the function will support it, but this version (that takes a vector) is currently the only one we need to do this with.

Definition at line 71 of file nnet-update-parallel.h.

References DoBackpropParallel().

Referenced by FastNnetCombiner::GetInitialModel().

                         {
   return DoBackpropParallel(nnet, minibatch_size, num_threads,
                             examples, num_frames, NULL);
 }

◆ ComputeObjfAndGradient() [1/3]

static BaseFloat kaldi::nnet2::ComputeObjfAndGradient	(	const std::vector< NnetExample > &	validation_set,
		const Vector< double > &	log_scale_params,
		const Nnet &	nnet,
		Vector< double > *	gradient
	)

static

Definition at line 25 of file shrink-nnet.cc.

References VectorBase< Real >::ApplyExp(), ComputeNnetGradient(), VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), Nnet::ScaleComponents(), and Nnet::SetZero().

                               {
   Vector<BaseFloat> scale_params(log_scale_params);
   scale_params.ApplyExp();
   Nnet nnet_scaled(nnet);
   nnet_scaled.ScaleComponents(scale_params);
   
   Nnet nnet_gradient(nnet);
   bool is_gradient = true;
   nnet_gradient.SetZero(is_gradient);
 
   // note: "ans" is normalized by the total weight of validation frames.
   int32 batch_size = 1024;
   BaseFloat ans = ComputeNnetGradient(nnet_scaled,
                                       validation_set,
                                       batch_size,
                                       &nnet_gradient);
 
   BaseFloat tot_count = validation_set.size();
   int32 i = 0; // index into log_scale_params.
   for (int32 j = 0; j < nnet_scaled.NumComponents(); j++) {
     const UpdatableComponent *uc =
         dynamic_cast<const UpdatableComponent*>(&(nnet.GetComponent(j))),
         *uc_gradient =
         dynamic_cast<const UpdatableComponent*>(&(nnet_gradient.GetComponent(j)));
     if (uc != NULL) {
       BaseFloat dotprod = uc->DotProduct(*uc_gradient) / tot_count;
       (*gradient)(i) = dotprod * scale_params(i); // gradient w.r.t log of scaling factor.
       // We multiply by scale_params(i) to take into account d/dx exp(x); "gradient"
       // is the gradient w.r.t. the log of the scale_params.
       i++;
     }
   }
   KALDI_ASSERT(i == log_scale_params.Dim());
   return ans;
 }

◆ ComputeObjfAndGradient() [2/3]

static BaseFloat kaldi::nnet2::ComputeObjfAndGradient	(	const std::vector< NnetExample > &	validation_set,
		const Vector< double > &	scale_params,
		const Nnet &	orig_nnet,
		const Nnet &	direction,
		Vector< double > *	gradient
	)

static

Definition at line 61 of file combine-nnet-a.cc.

References AddDirection(), ComputeNnetGradient(), VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, Nnet::NumComponents(), and Nnet::SetZero().

Referenced by CombineNnets(), CombineNnetsA(), ComputeObjfAndGradient(), FastNnetCombiner::FastNnetCombiner(), and ShrinkNnet().

                               {
   
   Vector<BaseFloat> scale_params_float(scale_params);
 
   Nnet nnet_combined;
   AddDirection(orig_nnet, direction, scale_params_float, &nnet_combined);
   
   Nnet nnet_gradient(nnet_combined);
   bool is_gradient = true;
   nnet_gradient.SetZero(is_gradient);
   
   // note: "ans" is normalized by the total weight of validation frames.
   int32 batch_size = 1024;
   BaseFloat ans = ComputeNnetGradient(nnet_combined,
                                       validation_set,
                                       batch_size,
                                       &nnet_gradient);
 
   BaseFloat tot_count = validation_set.size();
   int32 i = 0; // index into scale_params.
   for (int32 j = 0; j < nnet_combined.NumComponents(); j++) {
     const UpdatableComponent *uc_direction =
         dynamic_cast<const UpdatableComponent*>(&(direction.GetComponent(j))),
         *uc_gradient =
         dynamic_cast<const UpdatableComponent*>(&(nnet_gradient.GetComponent(j)));
     if (uc_direction != NULL) {
       BaseFloat dotprod = uc_direction->DotProduct(*uc_gradient) / tot_count;
       (*gradient)(i) = dotprod; 
       i++;
     }
   }
   KALDI_ASSERT(i == scale_params.Dim());
   return ans;
 }

◆ ComputeObjfAndGradient() [3/3]

static double kaldi::nnet2::ComputeObjfAndGradient	(	const std::vector< NnetExample > &	validation_set,
		const Vector< double > &	scale_params,
		const std::vector< Nnet > &	nnets,
		bool	debug,
		Vector< double > *	gradient
	)

static

Definition at line 124 of file combine-nnet.cc.

References CombineNnets(), ComputeNnetGradient(), ComputeObjfAndGradient(), VectorBase< Real >::Dim(), UpdatableComponent::DotProduct(), Nnet::GetComponent(), rnnlm::i, rnnlm::j, KALDI_ASSERT, KALDI_LOG, rnnlm::n, Nnet::NumComponents(), and Nnet::SetZero().

                               {
 
   Vector<BaseFloat> scale_params_float(scale_params);
 
   Nnet nnet_combined;
   CombineNnets(scale_params_float, nnets, &nnet_combined);
 
   Nnet nnet_gradient(nnet_combined);
   bool is_gradient = true;
   nnet_gradient.SetZero(is_gradient);
 
   // note: "ans" is normalized by the total weight of validation frames.
   int32 batch_size = 1024;
   double ans = ComputeNnetGradient(nnet_combined,
                                    validation_set,
                                    batch_size,
                                    &nnet_gradient);
 
   double tot_frames = validation_set.size();
   if (gradient != NULL) {
     int32 i = 0; // index into scale_params.
     for (int32 n = 0; n < static_cast<int32>(nnets.size()); n++) {
       for (int32 j = 0; j < nnet_combined.NumComponents(); j++) {
         const UpdatableComponent *uc =
             dynamic_cast<const UpdatableComponent*>(&(nnets[n].GetComponent(j))),
             *uc_gradient =
             dynamic_cast<const UpdatableComponent*>(&(nnet_gradient.GetComponent(j)));
         if (uc != NULL) {
           double dotprod = uc->DotProduct(*uc_gradient) / tot_frames;
           (*gradient)(i) = dotprod;
           i++;
         }
       }
     }
     KALDI_ASSERT(i == scale_params.Dim());
   }
 
   if (debug) {
     KALDI_LOG << "Double-checking gradient computation";
 
     Vector<BaseFloat> manual_gradient(scale_params.Dim());
     for (int32 i = 0; i < scale_params.Dim(); i++) {
       double delta = 1.0e-04, fg = fabs((*gradient)(i));
       if (fg < 1.0e-07) fg = 1.0e-07;
       if (fg * delta < 1.0e-05)
         delta = 1.0e-05 / fg;
 
       Vector<double> scale_params_temp(scale_params);
       scale_params_temp(i) += delta;
       double new_ans = ComputeObjfAndGradient(validation_set,
                                               scale_params_temp,
                                               nnets,
                                               false,
                                               NULL);
       manual_gradient(i) = (new_ans - ans) / delta;
     }
     KALDI_LOG << "Manually computed gradient is " << manual_gradient;
     KALDI_LOG << "Gradient we computed is " << *gradient;
   }
 
   return ans;
 }

◆ DoBackprop() [1/2]

double DoBackprop	(	const Nnet &	nnet,
		const std::vector< NnetExample > &	examples,
		Nnet *	nnet_to_update,
		double *	tot_accuracy = `NULL`
	)

This function computes the objective function and either updates the model or adds to parameter gradients.

Returns the cross-entropy objective function summed over all samples (normalize this by dividing by TotalNnetTrainingWeight(examples)). It is mostly a wrapper for a class NnetUpdater that's defined in nnet-update.cc, but we don't want to expose that complexity at this level. All these examples will be treated as one minibatch. If tot_accuracy != NULL, it outputs to that pointer the total (weighted) accuracy.

Definition at line 265 of file nnet-update.cc.

References NnetUpdater::ComputeForMinibatch(), ComputeNnetObjf(), Nnet::Info(), and KALDI_LOG.

Referenced by ComputeNnetGradient(), DoBackpropSingleThreaded(), FisherComputationClass::operator()(), DoBackpropParallelClass::operator()(), and TrainNnetSimple().

                                         {
   if (nnet_to_update == NULL)
     return ComputeNnetObjf(nnet, examples, tot_accuracy);
   try {
     NnetUpdater updater(nnet, nnet_to_update);
     return updater.ComputeForMinibatch(examples, tot_accuracy);
   } catch (...) {
     KALDI_LOG << "Error doing backprop, nnet info is: " << nnet.Info();
     throw;
   }
 }

◆ DoBackprop() [2/2]

double DoBackprop	(	const Nnet &	nnet,
		const std::vector< NnetExample > &	examples,
		Matrix< BaseFloat > *	examples_formatted,
		Nnet *	nnet_to_update,
		double *	tot_accuracy = `NULL`
	)

This version of DoBackprop allows you to separately call FormatNnetInput and provide the result to DoBackprop; this can be useful when using GPUs because the call to FormatNnetInput can be in a separate thread from the one that uses the GPU.

"examples_formatted" is really an input, but it's a pointer because internally we call Swap() on it, so we destroy its contents.

Definition at line 281 of file nnet-update.cc.

References NnetUpdater::ComputeForMinibatch(), ComputeNnetObjf(), Nnet::Info(), KALDI_LOG, and KALDI_WARN.

                                         {
   if (nnet_to_update == NULL) {
     KALDI_WARN << "Was not expecting to reach this code path "
                << "(wastefully formatting data twice)";
     return ComputeNnetObjf(nnet, examples, tot_accuracy);
  } try {
     NnetUpdater updater(nnet, nnet_to_update);
     return updater.ComputeForMinibatch(examples,
                                        examples_formatted,
                                        tot_accuracy);
   } catch (...) {
     KALDI_LOG << "Error doing backprop, nnet info is: " << nnet.Info();
     throw;
   }
 }

◆ DoBackpropParallel() [1/2]

double DoBackpropParallel	(	const Nnet &	nnet,
		int32	minibatch_size,
		SequentialNnetExampleReader *	example_reader,
		double *	tot_weight,
		Nnet *	nnet_to_update
	)

This function is similar to "DoBackprop" in nnet-update.h This function computes the objective function and either updates the model or computes parameter gradients.

It returns the cross-entropy objective function summed over all samples, weighted, and the total weight of the samples (typically the same as the #frames) into total_weight. It is mostly a wrapper for a class NnetUpdater that's defined in nnet-update.cc, but we don't want to expose that complexity at this level. Note: this function If &nnet == nnet_to_update, it assumes we're doing SGD and does something like Hogwild; otherwise it assumes we're computing a gradient and it sums up the gradients. The return value is the total log-prob summed over the #frames. It also outputs the #frames into "num_frames".

Definition at line 147 of file nnet-update-parallel.cc.

References ExamplesRepository::AcceptExamples(), DoBackpropSingleThreaded(), SequentialTableReader< Holder >::Done(), ExamplesRepository::ExamplesDone(), kaldi::g_num_threads, KALDI_LOG, SequentialTableReader< Holder >::Next(), and SequentialTableReader< Holder >::Value().

Referenced by ComputeNnetObjfParallel(), FastNnetCombiner::ComputeObjfAndGradient(), and main().

                                                 {
 #if HAVE_CUDA == 1
   // Our GPU code won't work with multithreading; we do this
   // to enable it to work with this code in the single-threaded
   // case.
   if (CuDevice::Instantiate().Enabled())
     return DoBackpropSingleThreaded(nnet, minibatch_size, examples_reader,
                                     tot_weight, nnet_to_update);
 #endif
 
   ExamplesRepository repository; // handles parallel programming issues regarding
   // the "examples" of data.
   double tot_log_prob = 0.0;
   *tot_weight = 0.0;
 
   // This function assumes you want the exact gradient, if
   // nnet_to_update != &nnet.
   const bool store_separate_gradients = (nnet_to_update != &nnet);
 
   DoBackpropParallelClass c(nnet, &repository, tot_weight,
                             &tot_log_prob, nnet_to_update,
                             store_separate_gradients);
 
   {
     // The initialization of the following class spawns the threads that
     // process the examples.  They get re-joined in its destructor.
     MultiThreader<DoBackpropParallelClass> m(g_num_threads, c);
 
     std::vector<NnetExample> examples;
     for (; !examples_reader->Done(); examples_reader->Next()) {
       examples.push_back(examples_reader->Value());
       if (examples.size() == minibatch_size)
         repository.AcceptExamples(&examples);
     }
     if (!examples.empty()) // partial minibatch.
       repository.AcceptExamples(&examples);
     // Here, the destructor of "m" re-joins the threads, and
     // does the summing of the gradients if we're doing gradient
     // computation (i.e. &nnet != nnet_to_update).  This gets
     // done in the destructors of the objects of type
     // DoBackpropParallelClass.
     repository.ExamplesDone();
   }
   KALDI_LOG << "Did backprop on " << *tot_weight << " examples, average log-prob "
             << "per frame is " << (tot_log_prob / *tot_weight);
   KALDI_LOG << "[this line is to be parsed by a script:] log-prob-per-frame="
             << (tot_log_prob / *tot_weight);
   return tot_log_prob;
 }

◆ DoBackpropParallel() [2/2]

double DoBackpropParallel	(	const Nnet &	nnet,
		int32	minibatch_size,
		int32	num_threads,
		const std::vector< NnetExample > &	examples,
		double *	num_frames,
		Nnet *	nnet_to_update
	)

This version of DoBackpropParallel takes a vector of examples, and will typically be used to compute the exact gradient.

Definition at line 221 of file nnet-update-parallel.cc.

References ExamplesRepository::AcceptExamples(), DoBackpropSingleThreaded(), ExamplesRepository::ExamplesDone(), and KALDI_VLOG.

                                                 {
   if (num_threads == 1) // support GPUs: special case for 1 thread.
     return DoBackpropSingleThreaded(nnet, minibatch_size, egs,
                                     tot_weight, nnet_to_update);
 
   ExamplesRepository repository; // handles parallel programming issues regarding
   // the "examples" of data.
   double tot_log_prob = 0.0;
   *tot_weight = 0;
   const bool store_separate_gradients = (nnet_to_update != &nnet);
 
   DoBackpropParallelClass c(nnet, &repository, tot_weight,
                             &tot_log_prob, nnet_to_update,
                             store_separate_gradients);
 
   {
     // The initialization of the following class spawns the threads that
     // process the examples.  They get re-joined in its destructor.
     MultiThreader<DoBackpropParallelClass> m(num_threads, c);
 
     int32 num_egs = egs.size();
     for (int32 offset = 0; offset < num_egs; offset += minibatch_size) {
       int32 this_minibatch_size = std::min(minibatch_size, num_egs - offset);
 
       // We waste a little time copying the examples here, but it's very minor.
       std::vector<NnetExample> examples(egs.begin() + offset,
                                                 egs.begin() + offset + this_minibatch_size);
 
       repository.AcceptExamples(&examples);
     }
 
     // Here, the destructor of "m" re-joins the threads, and
     // does the summing of the gradients if we're doing gradient
     // computation (i.e. &nnet != nnet_to_update).  This gets
     // done in the destructors of the objects of type
     // DoBackpropParallelClass.
     repository.ExamplesDone();
   }
   KALDI_VLOG(2) << "Did backprop on " << *tot_weight << " examples, average log-prob "
                 << "per frame is " << (tot_log_prob / *tot_weight);
   return tot_log_prob;
 }

◆ DoBackpropSingleThreaded()

double kaldi::nnet2::DoBackpropSingleThreaded	(	const Nnet &	nnet,
		int32	minibatch_size,
		const std::vector< NnetExample > &	egs,
		double *	tot_weight,
		Nnet *	nnet_to_update
	)

Definition at line 202 of file nnet-update-parallel.cc.

References DoBackprop(), rnnlm::i, and TotalNnetTrainingWeight().

Referenced by DoBackpropParallel().

                                                       {
   double ans = 0.0;
   *tot_weight = TotalNnetTrainingWeight(egs);
   for (size_t i = 0; i < egs.size(); i += minibatch_size) {
     std::vector<NnetExample>::const_iterator end_iter =
       (i + minibatch_size > egs.size() ? egs.end() :
        egs.begin() + i + minibatch_size);
     std::vector<NnetExample> this_egs(egs.begin() + i,
                                               end_iter);
     ans += DoBackprop(nnet, this_egs, nnet_to_update);
   }
   return ans;
 }

◆ ExampleToPdfPost()

void ExampleToPdfPost	(	const TransitionModel &	tmodel,
		const std::vector< int32 > &	silence_phones,
		std::string	criterion,
		bool	drop_frames,
		bool	one_silence_class,
		const DiscriminativeNnetExample &	eg,
		Posterior *	post
	)

Given a discriminative training example, this function works out posteriors at the pdf level (note: these are "discriminative-training posteriors" that may be positive or negative.

The denominator lattice "den_lat" in the example "eg" should already have had acoustic-rescoring done so that its acoustic probs are up to date, and any acoustic scaling should already have been applied.

"criterion" may be "mmi" or "mpfe" or "smbr". If criterion is "mmi", "drop_frames" means we don't include derivatives for frames where the numerator pdf is not in the denominator lattice.

if "one_silence_class" is true you can get a newer behavior for MPE/SMBR which will tend to reduce insertions.

"silence_phones" is a list of silence phones (this is only relevant for mpfe or smbr, if we want to treat silence specially).

Definition at line 838 of file nnet-example-functions.cc.

References fst::ConvertLattice(), kaldi::ConvertPosteriorToPdfs(), DiscriminativeNnetExample::den_lat, KALDI_ASSERT, kaldi::LatticeForwardBackwardMmi(), kaldi::LatticeForwardBackwardMpeVariants(), DiscriminativeNnetExample::num_ali, kaldi::ScalePosterior(), and DiscriminativeNnetExample::weight.

Referenced by SplitExampleStats::SplitExampleStats(), and UpdateHash().

                      {
   KALDI_ASSERT(criterion == "mpfe" || criterion == "smbr" || criterion == "mmi");
   
   Lattice lat;
   ConvertLattice(eg.den_lat, &lat);
   TopSort(&lat);
   if (criterion == "mpfe" || criterion == "smbr") {
     Posterior tid_post;
     LatticeForwardBackwardMpeVariants(tmodel, silence_phones, lat, eg.num_ali,
                                       criterion, one_silence_class, &tid_post);
     
     ConvertPosteriorToPdfs(tmodel, tid_post, post);
   } else {
     bool convert_to_pdf_ids = true, cancel = true;
     LatticeForwardBackwardMmi(tmodel, lat, eg.num_ali,
                               drop_frames, convert_to_pdf_ids, cancel,
                               post);
   }
   ScalePosterior(eg.weight, post);
 }

◆ ExciseDiscriminativeExample()

void ExciseDiscriminativeExample	(	const SplitDiscriminativeExampleConfig &	config,
		const TransitionModel &	tmodel,
		const DiscriminativeNnetExample &	eg,
		std::vector< DiscriminativeNnetExample > *	egs_out,
		SplitExampleStats *	stats_out
	)

Remove unnecessary frames from discriminative training example.

The output egs_out will be of size zero or one (usually one) after being called.

Definition at line 775 of file nnet-example-functions.cc.

References DiscriminativeExampleSplitter::Excise().

Referenced by main(), and SplitExampleStats::SplitExampleStats().

                                   {
   DiscriminativeExampleSplitter splitter(config, tmodel, eg, egs_out);
   splitter.Excise(stats_out);
 }

◆ ExpectOneOrTwoTokens()

static void kaldi::nnet2::ExpectOneOrTwoTokens	(	std::istream &	is,
		bool	binary,
		const std::string &	token1,
		const std::string &	token2
	)

static

Definition at line 135 of file nnet-component.cc.

References kaldi::ExpectToken(), KALDI_ASSERT, KALDI_ERR, and kaldi::ReadToken().

Referenced by NonlinearComponent::Read(), MaxoutComponent::Read(), MaxpoolingComponent::Read(), PnormComponent::Read(), PowerComponent::Read(), ScaleComponent::Read(), AffineComponent::Read(), AffineComponentPreconditioned::Read(), AffineComponentPreconditionedOnline::Read(), SpliceComponent::Read(), SpliceMaxComponent::Read(), BlockAffineComponent::Read(), BlockAffineComponentPreconditioned::Read(), SumGroupComponent::Read(), PermuteComponent::Read(), DctComponent::Read(), FixedLinearComponent::Read(), FixedAffineComponent::Read(), FixedScaleComponent::Read(), FixedBiasComponent::Read(), DropoutComponent::Read(), AdditiveNoiseComponent::Read(), and Convolutional1dComponent::Read().

                                                           {
   KALDI_ASSERT(token1 != token2);
   std::string temp;
   ReadToken(is, binary, &temp);
   if (temp == token1) {
     ExpectToken(is, binary, token2);
   } else {
     if (temp != token2) {
       KALDI_ERR << "Expecting token " << token1 << " or " << token2
                 << " but got " << temp;
     }
   }
 }

◆ FixNnet()

void FixNnet	(	const NnetFixConfig &	config,
		Nnet *	nnet
	)

Definition at line 31 of file nnet-fix.cc.

References AffineComponent::BiasParams(), count, NonlinearComponent::Count(), rnnlm::d, NonlinearComponent::DerivSum(), Nnet::GetComponent(), NonlinearComponent::InputDim(), KALDI_ASSERT, KALDI_LOG, KALDI_WARN, AffineComponent::LinearParams(), NnetFixConfig::max_average_deriv, NnetFixConfig::min_average_deriv, Nnet::NumComponents(), NnetFixConfig::parameter_factor, NnetFixConfig::relu_bias_change, and AffineComponent::SetParams().

Referenced by main(), and NnetFixConfig::Register().

                                                       {
   for (int32 c = 0; c + 1 < nnet->NumComponents(); c++) {
     AffineComponent *ac = dynamic_cast<AffineComponent*>(
         &(nnet->GetComponent(c)));
     NonlinearComponent *nc = dynamic_cast<NonlinearComponent*>(
         &(nnet->GetComponent(c + 1)));
     if (ac == NULL || nc == NULL) continue;
     // We only want to process this if it's of type SigmoidComponent
     // or TanhComponent.
     BaseFloat max_deriv; // The maximum derivative of this nonlinearity.
     bool is_relu = false;
     {
       SigmoidComponent *sc = dynamic_cast<SigmoidComponent*>(nc);
       TanhComponent *tc = dynamic_cast<TanhComponent*>(nc);
       RectifiedLinearComponent *rc = dynamic_cast<RectifiedLinearComponent*>(nc);
       if (sc != NULL) max_deriv = 0.25;
       else if (tc != NULL) max_deriv = 1.0;
       else if (rc != NULL) { max_deriv = 1.0; is_relu = true; }
       else continue; // E.g. SoftmaxComponent; we don't handle this.
     }
     double count = nc->Count();
     Vector<double> deriv_sum (nc->DerivSum());
     if (count == 0.0 || deriv_sum.Dim() == 0) {
       KALDI_WARN << "Cannot fix neural net because no statistics are stored.";
       continue;
     }
     Vector<BaseFloat> bias_params(ac->BiasParams());
     Matrix<BaseFloat> linear_params(ac->LinearParams());
     int32 dim = nc->InputDim(), num_small_deriv = 0, num_large_deriv = 0;
     for (int32 d = 0; d < dim; d++) {
       // deriv ratio is the ratio of the computed average derivative to the
       // maximum derivative of that nonlinear function.
       BaseFloat deriv_ratio = deriv_sum(d) / (count * max_deriv);
       KALDI_ASSERT(deriv_ratio >= 0.0 && deriv_ratio < 1.01); // Or there is an
                                                               // error in the
       // math.
       if (deriv_ratio < config.min_average_deriv) {
         // derivative is too small, meaning we've gone off into the "flat part"
         // of the sigmoid (or for ReLU, we're always-off).
         if (is_relu) {
           bias_params(d) += config.relu_bias_change;
         } else {
           BaseFloat parameter_factor = std::min(config.min_average_deriv /
                                                 deriv_ratio,
                                                 config.parameter_factor);
           // we need to reduce the parameters, so multiply by 1/parameter factor.
           bias_params(d) *= 1.0 / parameter_factor;
           linear_params.Row(d).Scale(1.0 / parameter_factor);
         }
         num_small_deriv++;
       } else if (deriv_ratio > config.max_average_deriv) {
         // derivative is too large, meaning we're only in the linear part of the
         // sigmoid, in the middle.  (or for ReLU, we're always-on.
         if (is_relu) {
           bias_params(d) -= config.relu_bias_change;
         } else {
           BaseFloat parameter_factor = std::min(deriv_ratio / config.max_average_deriv,
                                                 config.parameter_factor);
           // we need to increase the factors, so multiply by parameter_factor.
           bias_params(d) *= parameter_factor;
           linear_params.Row(d).Scale(parameter_factor);
         }
         num_large_deriv++;
       }
     }
     if (is_relu) {
       KALDI_LOG << "For layer " << c << " (ReLU units), increased bias for "
                 << num_small_deriv << " indexes and decreased it for "
                 << num_large_deriv << ", out of a total of " << dim;
     } else {
       KALDI_LOG << "For layer " << c << ", decreased parameters for "
                 << num_small_deriv << " indexes, and increased them for "
                 << num_large_deriv << " out of a total of " << dim;
     }
     ac->SetParams(bias_params, linear_params);
   }
 }

◆ FormatNnetInput()

void FormatNnetInput	(	const Nnet &	nnet,
		const std::vector< NnetExample > &	data,
		Matrix< BaseFloat > *	mat
	)

Takes the input to the nnet for a minibatch of examples, and formats as a single matrix.

data.size() must be > 0. Note: you will probably want to copy this to CuMatrix after you call this function. The num-rows of the output will, at exit, equal (1 + nnet.LeftContext() + nnet.RightContext()) * data.size(). The nnet is only needed so we can call LeftContext(), RightContext() and InputDim() on it.

Definition at line 207 of file nnet-update.cc.

References MatrixBase< Real >::CopyFromMat(), MatrixBase< Real >::CopyRowsFromVec(), Nnet::InputDim(), KALDI_ASSERT, kaldi::kUndefined, Nnet::LeftContext(), Matrix< Real >::Resize(), and Nnet::RightContext().

Referenced by NnetUpdater::FormatInput(), and NnetExampleBackgroundReader::ReadExamples().

                                                    {
   KALDI_ASSERT(data.size() > 0);
   int32 num_splice = 1 + nnet.RightContext() + nnet.LeftContext();
   KALDI_ASSERT(data[0].input_frames.NumRows() >= num_splice);
   
   int32 feat_dim = data[0].input_frames.NumCols(),
          spk_dim = data[0].spk_info.Dim(),
          tot_dim = feat_dim + spk_dim; // we append these at the neural net
                                        // input... note, spk_dim might be 0.
   KALDI_ASSERT(tot_dim == nnet.InputDim());
   KALDI_ASSERT(data[0].left_context >= nnet.LeftContext());
   int32 ignore_frames = data[0].left_context - nnet.LeftContext(); // If
   // the NnetExample has more left-context than we need, ignore some.
   // this may happen in settings where we increase the amount of context during
   // training, e.g. by adding layers that require more context.  
 
   int32 num_chunks = data.size();
   
   input_mat->Resize(num_splice * num_chunks,
                     tot_dim, kUndefined);
   
   for (int32 chunk = 0; chunk < num_chunks; chunk++) {
     SubMatrix<BaseFloat> dest(*input_mat,
                               chunk * num_splice, num_splice,
                               0, feat_dim);
 
     Matrix<BaseFloat> full_src(data[chunk].input_frames);
     SubMatrix<BaseFloat> src(full_src, ignore_frames, num_splice, 0, feat_dim);
                              
     dest.CopyFromMat(src);
     if (spk_dim != 0) {
       SubMatrix<BaseFloat> spk_dest(*input_mat,
                                     chunk * num_splice, num_splice,
                                     feat_dim, spk_dim);
       spk_dest.CopyRowsFromVec(data[chunk].spk_info);
     }
   }
 }

◆ GenRandomNnet()

Nnet * GenRandomNnet	(	int32	input_dim,
		int32	output_dim
	)

This function generates a random neural net, for testing purposes.

It will contain a random number of SigmoidComponent, AffineComponent and SpliceComponent, followed by a final AffineComponent and SoftmaxComponent. The parameters will all be randomly initialized.

Definition at line 772 of file nnet-nnet.cc.

References rnnlm::i, Nnet::Init(), AffineComponent::Init(), SpliceComponent::Init(), and Nnet::Nnet().

Referenced by UnitTestAmNnet(), UnitTestNnet(), UnitTestNnetCompute(), UnitTestNnetComputeChunked(), and UnitTestNnetDecodable().

                                       {
   std::vector<Component*> components;
   int32 cur_dim = input_dim;
   // have up to 10 layers before the final one.
   for (size_t i = 0; i < 10; i++) {
     if (rand() % 2 == 0) {
       // add an affine component.
       int32 next_dim = 50 + rand() % 100;
       BaseFloat learning_rate = 0.0001, param_stddev = 0.001,
           bias_stddev = 0.1;
       AffineComponent *component = new AffineComponent();
       component->Init(learning_rate, cur_dim, next_dim,
                       param_stddev, bias_stddev);
       components.push_back(component);
       cur_dim = next_dim;
     } else if (rand() % 2 == 0) {
       components.push_back(new SigmoidComponent(cur_dim));
     } else if (rand() % 2 == 0 && cur_dim < 200) {
       SpliceComponent *component = new SpliceComponent();
       std::vector<int32> context;
       while (true) {
         context.clear();
         for (int32 i = -3; i <= 3; i++) {
           if (rand() % 3 == 0)
             context.push_back(i);
         }
         if (!context.empty() && context.front() <= 0 &&
             context.back() >= 0)
           break;
       }
       component->Init(cur_dim, context);
       components.push_back(component);
       cur_dim = cur_dim * context.size();
     } else {
       break;
     }
   }
 
   {
     AffineComponent *component = new AffineComponent();
     BaseFloat learning_rate = 0.0001, param_stddev = 0.001,
         bias_stddev = 0.1;
     component->Init(learning_rate, cur_dim, output_dim,
                     param_stddev, bias_stddev);
     components.push_back(component);
     cur_dim = output_dim;
   }
 
   components.push_back(new SoftmaxComponent(cur_dim));
 
   Nnet *ans = new Nnet();
   ans->Init(&components);
   return ans;
 }

◆ GetCount()

int32 GetCount ( double expected_count )

Definition at line 31 of file nnet-copy-egs-discriminative.cc.

References KALDI_ASSERT, and kaldi::WithProb().

Referenced by main(), and ProcessFile().

                                       {
   KALDI_ASSERT(expected_count >= 0.0);
   int32 ans = 0;
   while (expected_count > 1.0) {
     ans++;
     expected_count--;
   }
   if (WithProb(expected_count))
     ans++;
   return ans;
 }

◆ GetInitialModel()

static int32 kaldi::nnet2::GetInitialModel	(	const std::vector< NnetExample > &	validation_set,
		const std::vector< Nnet > &	nnets
	)

static

Returns an integer saying which model to use: either 0 ...

num-models - 1 for the best individual model, or (#models) for the average of all of them.

Definition at line 49 of file combine-nnet.cc.

References CombineNnets(), ComputeNnetObjf(), KALDI_ASSERT, KALDI_LOG, rnnlm::n, and VectorBase< Real >::Set().

Referenced by FastNnetCombiner::FastNnetCombiner(), FastNnetCombiner::GetInitialParams(), and GetInitialScaleParams().

                                   {
   int32 minibatch_size = 1024;
   int32 num_nnets = static_cast<int32>(nnets.size());
   KALDI_ASSERT(!nnets.empty());
   BaseFloat tot_frames = validation_set.size();
   int32 best_n = -1;
   BaseFloat best_objf = -std::numeric_limits<BaseFloat>::infinity();
   Vector<BaseFloat> objfs(nnets.size());
   for (int32 n = 0; n < num_nnets; n++) {
     BaseFloat objf = ComputeNnetObjf(nnets[n], validation_set,
                                      minibatch_size) / tot_frames;
 
     if (n == 0 || objf > best_objf) {
       best_objf = objf;
       best_n = n;
     }
     objfs(n) = objf;
   }
   KALDI_LOG << "Objective functions for the source neural nets are " << objfs;
 
   int32 num_uc = nnets[0].NumUpdatableComponents();
 
   { // Now try a version where all the neural nets have the same weight.
     Vector<BaseFloat> scale_params(num_uc * num_nnets);
     scale_params.Set(1.0 / num_nnets);
     Nnet average_nnet;
     CombineNnets(scale_params, nnets, &average_nnet);
     BaseFloat objf = ComputeNnetObjf(average_nnet, validation_set,
                                      minibatch_size) / tot_frames;
     KALDI_LOG << "Objf with all neural nets averaged is " << objf;
     if (objf > best_objf) {
       return num_nnets;
     } else {
       return best_n;
     }
   }
 }

◆ GetInitialScaleParams()

static void kaldi::nnet2::GetInitialScaleParams	(	const NnetCombineConfig &	combine_config,
		const std::vector< NnetExample > &	validation_set,
		const std::vector< Nnet > &	nnets,
		Vector< double > *	scale_params
	)

static

Definition at line 91 of file combine-nnet.cc.

References GetInitialModel(), NnetCombineConfig::initial_model, KALDI_ASSERT, KALDI_LOG, Vector< Real >::Resize(), and VectorBase< Real >::Set().

Referenced by CombineNnets().

                                   {
 
   int32 initial_model = combine_config.initial_model,
       num_nnets = static_cast<int32>(nnets.size());
   if (initial_model < 0 || initial_model > num_nnets)
     initial_model = GetInitialModel(validation_set, nnets);
 
   KALDI_ASSERT(initial_model >= 0 && initial_model <= num_nnets);
   int32 num_uc = nnets[0].NumUpdatableComponents();
 
   scale_params->Resize(num_uc * num_nnets);
   if (initial_model < num_nnets) {
     KALDI_LOG << "Initializing with neural net with index " << initial_model;
     // At this point we're using the best of the individual neural nets.
     scale_params->Set(0.0);
 
     // Set the block of parameters corresponding to the "best" of the
     // source neural nets to
     SubVector<double> best_block(*scale_params, num_uc * initial_model, num_uc);
     best_block.Set(1.0);
   } else { // initial_model == num_nnets
     KALDI_LOG << "Initializing with all neural nets averaged.";
     scale_params->Set(1.0 / num_nnets);
   }
 }

◆ GetNnetStats()

void GetNnetStats	(	const NnetStatsConfig &	config,
		const Nnet &	nnet,
		std::vector< NnetStats > *	stats
	)

Definition at line 99 of file nnet-stats.cc.

References NnetStatsConfig::bucket_width, Nnet::GetComponent(), KALDI_ASSERT, NnetStats::NnetStats(), and Nnet::NumComponents().

                                                {
   KALDI_ASSERT(stats->size() == 0);
   for (int32 c = 0; c + 1 < nnet.NumComponents(); c++) {
     const AffineComponent *ac = dynamic_cast<const AffineComponent*>(
         &(nnet.GetComponent(c)));
     if (ac == NULL) continue;
     const NonlinearComponent *nc = dynamic_cast<const NonlinearComponent*>(
         &(nnet.GetComponent(c + 1)));
     if (nc == NULL) continue;
     // exclude softmax.
     const SoftmaxComponent *sc = dynamic_cast<const SoftmaxComponent*>(
         &(nnet.GetComponent(c + 1)));
     if (sc != NULL) continue;
     stats->push_back(NnetStats(c, config.bucket_width));
     stats->back().AddStatsFromNnet(nnet);
   }
 }

◆ GetUpdateDirection()

static void kaldi::nnet2::GetUpdateDirection	(	const std::vector< Nnet > &	nnets,
		Nnet *	direction
	)

static

Definition at line 31 of file combine-nnet-a.cc.

References Nnet::AddNnet(), KALDI_ASSERT, rnnlm::n, kaldi::nnet3::NumUpdatableComponents(), Nnet::ScaleComponents(), and VectorBase< Real >::Set().

Referenced by CombineNnetsA().

                                                 {
   KALDI_ASSERT(nnets.size() > 1);
   int32 num_new_nnets = nnets.size() - 1;
   Vector<BaseFloat> scales(nnets[0].NumUpdatableComponents());
 
   scales.Set(1.0 / num_new_nnets);
   
   *direction = nnets[1];
   direction->ScaleComponents(scales); // first of the new nnets.
   for (int32 n = 2; n < 1 + num_new_nnets; n++)
     direction->AddNnet(scales, nnets[n]);
   // now "direction" is the average of the new nnets.  Subtract
   // the old nnet's parameters.
   scales.Set(-1.0);
   direction->AddNnet(scales, nnets[0]);
 }

◆ GiveNnetCorrectTopology()

static void kaldi::nnet2::GiveNnetCorrectTopology	(	Nnet *	nnet,
		AffineComponent **	affine_component,
		SoftmaxComponent **	softmax_component,
		SumGroupComponent **	sum_group_component
	)

static

This function makes sure the neural net ends with a SumGroupComponent.

If it doesn't, it adds one (with a single mixture/matrix corresponding to each output element.) [Before doing so, it makes sure that the last layer is a SoftmaxLayer, which is what we expect. You can remove this check if there is some use-case that makes sense where the type of the previous layer is different.

Definition at line 37 of file mixup-nnet.cc.

References Nnet::Append(), Nnet::GetComponent(), KALDI_ASSERT, KALDI_ERR, KALDI_LOG, Nnet::NumComponents(), Component::OutputDim(), and Component::Type().

Referenced by MixupNnet().

                                                                              {
   int32 nc = nnet->NumComponents();
   KALDI_ASSERT(nc > 0);
   Component* component = &(nnet->GetComponent(nc - 1));
   if ((*sum_group_component =
        dynamic_cast<SumGroupComponent*>(component)) == NULL) {
     KALDI_LOG << "Adding SumGroupComponent to neural net.";
     int32 dim = component->OutputDim();
     // Give it the same learning rate as the first updatable layer we have.
     std::vector<int32> sizes(dim, 1); // a vector of all ones, of dimension "dim".
   
     *sum_group_component = new SumGroupComponent();
     (*sum_group_component)->Init(sizes);
     nnet->Append(*sum_group_component);
     nc++;
   }
   component = &(nnet->GetComponent(nc - 2));    
   if ((*softmax_component = dynamic_cast<SoftmaxComponent*>(component)) == NULL)
     KALDI_ERR << "Neural net has wrong topology: expected second-to-last "
               << "component to be SoftmaxComponent, type is "
               << component->Type();
   component = &(nnet->GetComponent(nc - 3));
   if ((*affine_component = dynamic_cast<AffineComponent*>(component)) == NULL)
     KALDI_ERR << "Neural net has wrong topology: expected third-to-last "
               << "component to be AffineComponent, type is "
               << component->Type();
 }

◆ HasSimpleLabels()

bool kaldi::nnet2::HasSimpleLabels	(	const NnetExample &	eg,
		std::vector< int32 > *	simple_labels
	)

Definition at line 32 of file nnet-example.cc.

References NnetExample::labels.

Referenced by NnetExample::Write().

                                      {
   size_t num_frames = eg.labels.size();
   for (int32 t = 0; t < num_frames; t++)
     if (eg.labels[t].size() != 1 || eg.labels[t][0].second != 1.0)
       return false;
   simple_labels->resize(num_frames);
   for (int32 t = 0; t < num_frames; t++)
     (*simple_labels)[t] = eg.labels[t][0].first;
   return true;
 }

◆ IndexOfSoftmaxLayer()

int32 IndexOfSoftmaxLayer ( const Nnet & nnet )

If "nnet" has exactly one softmax layer, this function will return its index; otherwise it will return -1.

Definition at line 27 of file nnet-functions.cc.

References Nnet::GetComponent(), and Nnet::NumComponents().

Referenced by main().

                                             {
   int32 index = -1, nc = nnet.NumComponents();
   for (int32 c = 0; c < nc; c++) {
     const Component *component = &(nnet.GetComponent(c));
     if (dynamic_cast<const SoftmaxComponent*>(component) != NULL) {
       if (index != -1) return -1; // >1 softmax components.
       else index = c;
     }
   }
   return index;
 }

◆ InsertComponents()

void InsertComponents	(	const Nnet &	src_nnet,
		int32	c,
		Nnet *	dest_nnet
	)

Inserts the components of one neural network into a particular place in the other one.

This is useful for adding hidden layers to a neural net. Inserts the components of "src_nnet" before component index c of "dest_nnet".

Definition at line 39 of file nnet-functions.cc.

References Component::Copy(), Nnet::GetComponent(), Nnet::Init(), KALDI_ASSERT, and Nnet::NumComponents().

Referenced by main().

                                        {
   KALDI_ASSERT(c_to_insert >= 0 && c_to_insert <= dest_nnet->NumComponents());
   int32 c_tot = dest_nnet->NumComponents() + src_nnet.NumComponents();
   std::vector<Component*> components(c_tot);
   for (int32 c = 0; c < c_to_insert; c++)
     components[c] = dest_nnet->GetComponent(c).Copy();
   for (int32 c = 0; c < src_nnet.NumComponents(); c++)
     components[c + c_to_insert] = src_nnet.GetComponent(c).Copy();
   for (int32 c = c_to_insert; c < dest_nnet->NumComponents(); c++)
     components[c + src_nnet.NumComponents()] = dest_nnet->GetComponent(c).Copy();
   // Re-initialize "dest_nnet" from the resulting list of components.
 
   // The Init method will take ownership of the pointers in the vector:
   dest_nnet->Init(&components);
 }

◆ KlDivergence()

BaseFloat kaldi::nnet2::KlDivergence	(	const Vector< BaseFloat > &	p,
		const Vector< BaseFloat > &	q
	)

Definition at line 31 of file nnet-adjust-priors.cc.

References VectorBase< Real >::Dim(), rnnlm::i, KALDI_ASSERT, KALDI_WARN, kaldi::Log(), and VectorBase< Real >::Sum().

Referenced by PrintPriorDiagnostics().

                                                    {
   BaseFloat sum_p = p.Sum(), sum_q = q.Sum();
   if (fabs(sum_p - 1.0) > 0.01 || fabs(sum_q - 1.0) > 0.01) {
     KALDI_WARN << "KlDivergence: vectors are not close to being normalized "
                << sum_p << ", " << sum_q;
   }
   KALDI_ASSERT(p.Dim() == q.Dim());
   double ans = 0.0;
 
   for (int32 i = 0; i < p.Dim(); i++) {
     BaseFloat p_prob = p(i) / sum_p, q_prob = q(i) / sum_q;
     ans += p_prob * Log(p_prob / q_prob);
   }
   return ans;
 }

◆ LatticeToDiscriminativeExample()

bool LatticeToDiscriminativeExample	(	const std::vector< int32 > &	alignment,
		const Matrix< BaseFloat > &	feats,
		const CompactLattice &	clat,
		BaseFloat	weight,
		int32	left_context,
		int32	right_context,
		DiscriminativeNnetExample *	eg
	)

Converts lattice to discriminative training example.

returns true on success, false on failure such as mismatched input (will also warn in this case).

Definition at line 27 of file nnet-example-functions.cc.

References DiscriminativeNnetExample::Check(), kaldi::CompactLatticeStateTimes(), DiscriminativeNnetExample::den_lat, DiscriminativeNnetExample::input_frames, KALDI_ASSERT, KALDI_WARN, DiscriminativeNnetExample::left_context, DiscriminativeNnetExample::num_ali, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), MatrixBase< Real >::Range(), Matrix< Real >::Resize(), MatrixBase< Real >::Row(), and DiscriminativeNnetExample::weight.

Referenced by main(), and SplitExampleStats::SplitExampleStats().

                                    {
   KALDI_ASSERT(left_context >= 0 && right_context >= 0);
   int32 num_frames = alignment.size();
   if (num_frames == 0) {
     KALDI_WARN << "Empty alignment";
     return false;
   }
   if (num_frames != feats.NumRows()) {
     KALDI_WARN << "Dimension mismatch: alignment " << num_frames
                << " versus feats " << feats.NumRows();
     return false;
   }
   std::vector<int32> times;
   int32 num_frames_clat = CompactLatticeStateTimes(clat, &times);  
   if (num_frames_clat != num_frames) {
     KALDI_WARN << "Numerator/frames versus denlat frames mismatch: "
                << num_frames << " versus " << num_frames_clat;
     return false;
   }
   eg->weight = weight;
   eg->num_ali = alignment;
   eg->den_lat = clat;
 
   int32 feat_dim = feats.NumCols();
   eg->input_frames.Resize(left_context + num_frames + right_context,
                           feat_dim);
   eg->input_frames.Range(left_context, num_frames,
                          0, feat_dim).CopyFromMat(feats);
 
   // Duplicate the first and last frames.
   for (int32 t = 0; t < left_context; t++)
     eg->input_frames.Row(t).CopyFromVec(feats.Row(0));
   for (int32 t = 0; t < right_context; t++)
     eg->input_frames.Row(left_context + num_frames + t).CopyFromVec(
         feats.Row(num_frames - 1));
 
   eg->left_context = left_context;
   eg->Check();
   return true;
 }

◆ LimitRankParallel()

void LimitRankParallel	(	const NnetLimitRankOpts &	opts,
		Nnet *	nnet
	)

This function limits the rank of each affine transform in the neural net, by zeroing out the smallest singular values.

The number of singular values to zero out is determined on a layer by layer basis, using "parameter_proportion" to set the proportion of parameters to remove.

Definition at line 99 of file nnet-limit-rank.cc.

References Nnet::GetComponent(), NnetLimitRankOpts::num_threads, TaskSequencerConfig::num_threads, Nnet::NumComponents(), and TaskSequencer< C >::Run().

Referenced by NnetLimitRankOpts::Register().

                                         {
   TaskSequencerConfig task_config;
   task_config.num_threads = opts.num_threads;
   TaskSequencer<LimitRankClass> tc(task_config);
   for (int32 c = 0; c < nnet->NumComponents(); c++) {
     if (dynamic_cast<AffineComponent*>(&(nnet->GetComponent(c))) != NULL)
       tc.Run(new LimitRankClass(opts, c, nnet));
   }
 }

◆ MakePair()

static Int32Pair kaldi::nnet2::MakePair	(	int32	first,
		int32	second
	)

inlinestatic

Definition at line 27 of file train-nnet-ensemble.cc.

References Int32Pair::first, and Int32Pair::second.

Referenced by NnetEnsembleTrainer::TrainOneMinibatch().

                                                             {
   Int32Pair ans;
   ans.first = first;
   ans.second = second;
   return ans;
 }

◆ MixupNnet()

void MixupNnet	(	const NnetMixupConfig &	mixup_config,
		Nnet *	nnet
	)

This function works as follows.

This function does something similar to Gaussian mixture splitting for GMMs, except applied to the output layer of the neural network.

We first make sure the neural net has the correct topology, so its last component is a SumGroupComponent.

We then get the counts for each matrix in the SumGroupComponent (these will either correspond to leaves in the decision tree, or level-1 leaves, if we have a 2-level-tree system). We work out the total count for each of these matrices, by getting the count from the SoftmaxComponent.

We then increase, if necessary, the dimensions that the SumGroupComponent sums over increase the dimension of the SoftmaxComponent if necessary, and duplicate and then perturb the relevant rows of the AffineComponent.

We create additional outputs, which will be summed over using a SumGroupComponent.

Definition at line 86 of file mixup-nnet.cc.

References Nnet::Check(), GiveNnetCorrectTopology(), NnetMixupConfig::min_count, SoftmaxComponent::MixUp(), NnetMixupConfig::num_mixtures, NnetMixupConfig::perturb_stddev, and NnetMixupConfig::power.

Referenced by main(), and NnetMixupConfig::Register().

                            {
   AffineComponent *affine_component = NULL;
   SoftmaxComponent *softmax_component = NULL;
   SumGroupComponent *sum_group_component = NULL;
   GiveNnetCorrectTopology(nnet,
                           &affine_component,
                           &softmax_component,
                           &sum_group_component); // Adds a SumGroupComponent if needed.
   
   softmax_component->MixUp(mixup_config.num_mixtures,
                            mixup_config.power,
                            mixup_config.min_count,
                            mixup_config.perturb_stddev,
                            affine_component,
                            sum_group_component);
   nnet->Check(); // Checks that dimensions all match up.
 }

◆ NnetComputation()

void NnetComputation	(	const Nnet &	nnet,
		const CuMatrixBase< BaseFloat > &	input,
		bool	pad_input,
		CuMatrixBase< BaseFloat > *	output
	)

Does the basic neural net computation, on a sequence of data (e.g.

an utterance). If pad_input==true we'll pad the input with enough frames of context, and the output will be a matrix of #frames by the output-dim of the network, typically representing state-level posteriors. If pad_input==false we won't do this and the output will have a lower #frames than the input; we lose nnet.LeftContext() at the left and nnet.RightContext() at the output.

Definition at line 160 of file nnet-compute.cc.

References CuMatrixBase< Real >::CopyFromMat(), NnetComputer::GetOutput(), and NnetComputer::Propagate().

Referenced by NnetComputation::Clear(), CachingOptimizingCompiler::CompileNoShortcut(), CachingOptimizingCompiler::CompileViaShortcut(), DecodableAmNnetParallel::Compute(), DecodableNnet2Online::ComputeForFrame(), DecodableAmNnet::DecodableAmNnet(), main(), ComputationCache::Read(), UnitTestNnetCompute(), and UnitTestNnetComputeChunked().

                                                       {
   NnetComputer nnet_computer(nnet, input, pad_input, NULL);
   nnet_computer.Propagate();
   output->CopyFromMat(nnet_computer.GetOutput());
 }

◆ NnetComputationChunked()

void NnetComputationChunked	(	const Nnet &	nnet,
		const CuMatrixBase< BaseFloat > &	input,
		int32	chunk_size,
		CuMatrixBase< BaseFloat > *	output
	)

Does the basic neural net computation, on a sequence of data (e.g.

an utterance). This variant of NnetComputation chunks the input according to chunk_size and does the posterior computation chunk by chunk. This allows the computation to be performed on the GPU when the input matrix is very large. Input is padded with enough frames of context so that the output will be a matrix with input.NumRows().

Definition at line 169 of file nnet-compute.cc.

References CuMatrixBase< Real >::CopyFromMat(), NnetComputer::GetOutput(), rnnlm::i, Nnet::LeftContext(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), NnetComputer::Propagate(), CuMatrixBase< Real >::Range(), CuMatrix< Real >::Resize(), Nnet::RightContext(), and CuMatrixBase< Real >::Row().

Referenced by main(), and UnitTestNnetComputeChunked().

                                                       {
   int32 num_rows,
        num_chunks = ceil((BaseFloat)input.NumRows() / chunk_size),
        dim = input.NumCols(),
        left_context = nnet.LeftContext(),
        right_context = nnet.RightContext();
   CuMatrix<BaseFloat> full_input;
   num_rows = left_context + input.NumRows() + right_context;
   full_input.Resize(num_rows, dim);
   full_input.Range(left_context, input.NumRows(),
             0, dim).CopyFromMat(input);
   for (int32 i = 0; i < left_context; i++)
     full_input.Row(i).CopyFromVec(input.Row(0));
   int32 last_row = input.NumRows() - 1;
   for (int32 i = 0; i < right_context; i++)
     full_input.Row(num_rows - i - 1).CopyFromVec(input.Row(last_row));
 
   for (int32 i = 0; i < num_chunks; i++) {
     int32 index = i * chunk_size,
           offset = std::min(num_rows - chunk_size * i, 
                             left_context + chunk_size + right_context);
     CuSubMatrix<BaseFloat> chunk_input(full_input, index, offset, 0, dim);
     CuMatrix<BaseFloat> cu_chunk_input(chunk_input);
 
     // Note: we have already accounted for input padding, so we pass
     // pad_input==false to the NnetComputer.
     NnetComputer nnet_computer(nnet, cu_chunk_input, false, NULL);
     nnet_computer.Propagate();
     CuMatrix<BaseFloat> cu_chunk_output(nnet_computer.GetOutput());
     CuSubMatrix<BaseFloat> chunk_out(*output, i * chunk_size, 
                            cu_chunk_output.NumRows(), 0, 
                            cu_chunk_output.NumCols());
     chunk_out.CopyFromMat(cu_chunk_output);
   }
 }

◆ NnetDiscriminativeUpdate()

void NnetDiscriminativeUpdate	(	const AmNnet &	am_nnet,
		const TransitionModel &	tmodel,
		const NnetDiscriminativeUpdateOptions &	opts,
		const DiscriminativeNnetExample &	eg,
		Nnet *	nnet_to_update,
		NnetDiscriminativeStats *	stats
	)

Does the neural net computation, lattice forward-backward, and backprop, for either the MMI, MPFE or SMBR objective functions.

If nnet_to_update == &(am_nnet.GetNnet()), then this does stochastic gradient descent, otherwise (assuming you have called SetZero(true) on *nnet_to_update) it will compute the gradient on this data. If nnet_to_update_ == NULL, no backpropagation is done.

Note: we ignore any existing acoustic score in the lattice of "eg".

For display purposes you should normalize the sum of this return value by dividing by the sum over the examples, of the number of frames (num_ali.size()) times the weight.

Something you need to be careful with is that the occupation counts and the derivative are, following tradition, missing a factor equal to the acoustic scale. So you need to multiply them by that scale if you plan to do something like L-BFGS in which you look at both the derivatives and function values.

Definition at line 365 of file nnet-compute-discriminative.cc.

References NnetDiscriminativeUpdater::Update().

Referenced by main(), NnetDiscriminativeStats::NnetDiscriminativeStats(), and DiscTrainParallelClass::operator()().

                                                               {
   NnetDiscriminativeUpdater updater(am_nnet, tmodel, opts, eg,
                                     nnet_to_update, stats);
   updater.Update();
 }

◆ NnetDiscriminativeUpdateParallel()

void NnetDiscriminativeUpdateParallel	(	const AmNnet &	am_nnet,
		const TransitionModel &	tmodel,
		const NnetDiscriminativeUpdateOptions &	opts,
		int32	num_threads,
		SequentialDiscriminativeNnetExampleReader *	example_reader,
		Nnet *	nnet_to_update,
		NnetDiscriminativeStats *	stats
	)

Definition at line 189 of file nnet-compute-discriminative-parallel.cc.

References DiscriminativeExamplesRepository::AcceptExample(), NnetDiscriminativeUpdateOptions::criterion, SequentialTableReader< Holder >::Done(), DiscriminativeExamplesRepository::ExamplesDone(), AmNnet::GetNnet(), SequentialTableReader< Holder >::Next(), NnetDiscriminativeStats::Print(), and SequentialTableReader< Holder >::Value().

Referenced by main().

                                     {
 
   DiscriminativeExamplesRepository repository;
 
   const bool store_separate_gradients = (nnet_to_update != &(am_nnet.GetNnet()));
 
   DiscTrainParallelClass c(am_nnet, tmodel, opts,
                            store_separate_gradients,
                            &repository, nnet_to_update, stats);
 
   {
     // The initialization of the following class spawns the threads that
     // process the examples.  They get re-joined in its destructor.
     MultiThreader<DiscTrainParallelClass> m(num_threads, c);
 
     for (; !example_reader->Done(); example_reader->Next()) {
       repository.AcceptExample(example_reader->Value());
     }
     repository.ExamplesDone();
   }
   stats->Print(opts.criterion);
 }

◆ NnetGradientComputation() [1/2]

BaseFloat kaldi::nnet2::NnetGradientComputation	(	const Nnet &	nnet,
		const MatrixBase< BaseFloat > &	input,
		bool	pad_input,
		BaseFloat	utterance_weight,
		const std::vector< int32 > &	labels,
		Nnet *	nnet_to_update
	)

Does the neural net computation and backprop, given input and labels.

Note: if pad_input==true the number of rows of input should be the same as the number of labels, and if false, you should omit nnet.LeftContext() labels on the left and nnet.RightContext() on the right. If nnet_to_update == &nnet, then this does stochastic gradient descent, otherwise (assuming you have called SetZero(true) on *nnet_to_update) it will compute the gradient on this data. Returns the total objective function summed over the frames, times the utterance weight.

◆ NnetGradientComputation() [2/2]

BaseFloat kaldi::nnet2::NnetGradientComputation	(	const Nnet &	nnet,
		const CuMatrixBase< BaseFloat > &	input,
		bool	pad_input,
		const Posterior &	pdf_post,
		Nnet *	nnet_to_update
	)

Definition at line 208 of file nnet-compute.cc.

References NnetComputer::Backprop(), NnetComputer::ComputeLastLayerDeriv(), and NnetComputer::Propagate().

                                                         {
   NnetComputer nnet_computer(nnet, input, pad_input, nnet_to_update);
   nnet_computer.Propagate();
   CuMatrix<BaseFloat> deriv;
   BaseFloat ans;
   ans = nnet_computer.ComputeLastLayerDeriv(pdf_post, &deriv);  
   nnet_computer.Backprop(&deriv);
   return ans;
 }

◆ ParseFromString() [1/5]

bool ParseFromString	(	const std::string &	name,
		std::string *	string,
		int32 *	param
	)

Functions used in Init routines.

Suppose name=="foo", if "string" has a field like foo=12, this function will set "param" to 12 and remove that element from "string". It returns true if the parameter was read.

Definition at line 153 of file nnet-component.cc.

References kaldi::ConvertStringToInteger(), rnnlm::i, rnnlm::j, KALDI_ERR, and kaldi::SplitStringToVector().

                                    {
   std::vector<std::string> split_string;
   SplitStringToVector(*string, " \t", true,
                       &split_string);
   std::string name_equals = name + "="; // the name and then the equals sign.
   size_t len = name_equals.length();
 
   for (size_t i = 0; i < split_string.size(); i++) {
     if (split_string[i].compare(0, len, name_equals) == 0) {
       if (!ConvertStringToInteger(split_string[i].substr(len), param))
         KALDI_ERR << "Bad option " << split_string[i];
       *string = "";
       // Set "string" to all the pieces but the one we used.
       for (size_t j = 0; j < split_string.size(); j++) {
         if (j != i) {
           if (!string->empty()) *string += " ";
           *string += split_string[j];
         }
       }
       return true;
     }
   }
   return false;
 }

◆ ParseFromString() [2/5]

bool ParseFromString	(	const std::string &	name,
		std::string *	string,
		bool *	param
	)

This version is for parameters of type bool, which can appear as any string beginning with f, F, t or T.

Definition at line 179 of file nnet-component.cc.

References rnnlm::i, rnnlm::j, KALDI_ERR, and kaldi::SplitStringToVector().

                                   {
   std::vector<std::string> split_string;
   SplitStringToVector(*string, " \t", true,
                       &split_string);
   std::string name_equals = name + "="; // the name and then the equals sign.
   size_t len = name_equals.length();
 
   for (size_t i = 0; i < split_string.size(); i++) {
     if (split_string[i].compare(0, len, name_equals) == 0) {
       std::string b = split_string[i].substr(len);
       if (b.empty())
         KALDI_ERR << "Bad option " << split_string[i];
       if (b[0] == 'f' || b[0] == 'F') *param = false;
       else if (b[0] == 't' || b[0] == 'T') *param = true;
       else
         KALDI_ERR << "Bad option " << split_string[i];
       *string = "";
       // Set "string" to all the pieces but the one we used.
       for (size_t j = 0; j < split_string.size(); j++) {
         if (j != i) {
           if (!string->empty()) *string += " ";
           *string += split_string[j];
         }
       }
       return true;
     }
   }
   return false;
 }

◆ ParseFromString() [3/5]

bool ParseFromString	(	const std::string &	name,
		std::string *	string,
		BaseFloat *	param
	)

This version is for parameters of type BaseFloat.

Definition at line 210 of file nnet-component.cc.

References kaldi::ConvertStringToReal(), rnnlm::i, rnnlm::j, KALDI_ERR, and kaldi::SplitStringToVector().

                                        {
   std::vector<std::string> split_string;
   SplitStringToVector(*string, " \t", true,
                       &split_string);
   std::string name_equals = name + "="; // the name and then the equals sign.
   size_t len = name_equals.length();
 
   for (size_t i = 0; i < split_string.size(); i++) {
     if (split_string[i].compare(0, len, name_equals) == 0) {
       if (!ConvertStringToReal(split_string[i].substr(len), param))
         KALDI_ERR << "Bad option " << split_string[i];
       *string = "";
       // Set "string" to all the pieces but the one we used.
       for (size_t j = 0; j < split_string.size(); j++) {
         if (j != i) {
           if (!string->empty()) *string += " ";
           *string += split_string[j];
         }
       }
       return true;
     }
   }
   return false;
 }

◆ ParseFromString() [4/5]

bool kaldi::nnet2::ParseFromString	(	const std::string &	name,
		std::string *	string,
		std::string *	param
	)

Definition at line 236 of file nnet-component.cc.

References rnnlm::i, rnnlm::j, and kaldi::SplitStringToVector().

                                        {
   std::vector<std::string> split_string;
   SplitStringToVector(*string, " \t", true,
                       &split_string);
   std::string name_equals = name + "="; // the name and then the equals sign.
   size_t len = name_equals.length();
 
   for (size_t i = 0; i < split_string.size(); i++) {
     if (split_string[i].compare(0, len, name_equals) == 0) {
       *param = split_string[i].substr(len);
 
       // Set "string" to all the pieces but the one we used.
       *string = "";
       for (size_t j = 0; j < split_string.size(); j++) {
         if (j != i) {
           if (!string->empty()) *string += " ";
           *string += split_string[j];
         }
       }
       return true;
     }
   }
   return false;
 }

◆ ParseFromString() [5/5]

bool ParseFromString	(	const std::string &	name,
		std::string *	string,
		std::vector< int32 > *	param
	)

This version is for parameters of type std::vector<int32>; it expects them as a colon-separated list, without spaces.

Definition at line 262 of file nnet-component.cc.

References rnnlm::i, rnnlm::j, KALDI_ERR, kaldi::SplitStringToIntegers(), and kaldi::SplitStringToVector().

                                               {
   std::vector<std::string> split_string;
   SplitStringToVector(*string, " \t", true,
                       &split_string);
   std::string name_equals = name + "="; // the name and then the equals sign.
   size_t len = name_equals.length();
 
   for (size_t i = 0; i < split_string.size(); i++) {
     if (split_string[i].compare(0, len, name_equals) == 0) {
       if (!SplitStringToIntegers(split_string[i].substr(len), ":",
                                  false, param))
         KALDI_ERR << "Bad option " << split_string[i];
       *string = "";
       // Set "string" to all the pieces but the one we used.
       for (size_t j = 0; j < split_string.size(); j++) {
         if (j != i) {
           if (!string->empty()) *string += " ";
           *string += split_string[j];
         }
       }
       return true;
     }
   }
   return false;
 }

◆ PreconditionDirections()

void PreconditionDirections	(	const CuMatrixBase< BaseFloat > &	R,
		double	lambda,
		CuMatrixBase< BaseFloat > *	P
	)

See below for comment.

The function PreconditionDirections views the input R as a set of directions or gradients, each row r_i being one of the directions.

For each i it constructs a preconditioning matrix G_i formed from the *other* i's, using the formula:

G_i = ( I + (1/(N-1)) {j i} r_j r_j^T)^{-1},

where N is the number of rows in R. This can be seen as a kind of estimated Fisher matrix that has been smoothed with the identity to make it invertible. We recommend that you set using: = /(N D) trace(R^T, R) for some small such as = 0.1. However, we leave this to the caller because there are reasons relating to unbiased-ness of the resulting stochastic gradient descent, why you might want to set using "other" data, e.g. a previous minibatch.

The output of this function is a matrix P, each row p_i of which is related to r_i by: p_i = G_i r_i Here, p_i is preconditioned by an estimated Fisher matrix in such a way that it's suitable to be used as an update direction.

Definition at line 26 of file nnet-precondition.cc.

Referenced by PreconditionDirectionsAlpha(), PreconditionDirectionsAlphaRescaled(), and UnitTestPreconditionDirections().

                                                         {
   
   int32 N = R.NumRows(), D = R.NumCols();
   KALDI_ASSERT(SameDim(R, *P) && N > 0);
   if (N == 1) {
     KALDI_WARN << "Trying to precondition set of only one frames: returning "
                << "unchanged.  Ignore this warning if infrequent.";
     P->CopyFromMat(R);
     return;
   }
   CuMatrixBase<BaseFloat> &Q = *P;
   
   if (N >= D) {
     // Compute G = (\lambda I + 1/(N-1) R^T R)^{-1} by direct inversion.
     // G <-- lambda I.
     CuMatrix<BaseFloat> G(D, D);
     G.AddToDiag(lambda);
     // G += 1.0/(N-1) * R^T R.
     G.SymAddMat2(1.0 / (N-1), R, kTrans, 1.0);
     G.CopyLowerToUpper();
     if (GetVerboseLevel() >= 5 && Rand() % 20 == 0) {
       CuSpMatrix<BaseFloat> tmp(G, kTakeLower);
       SpMatrix<BaseFloat> G_cpu(tmp);
       G_cpu.PrintEigs("G");
     }
     G.SymInvertPosDef();
     // Q <-- R G^T (we just make it transposed as we think
     // it will be slightly faster; it's symmetric).
     Q.AddMatMat(1.0, R, kNoTrans, G, kTrans, 0.0);
   } else {
     // Through a lot of rearrangements, it turns out that
     // if we let  S = (\lambda I + 1/(N-1) R R^T)^{-1}
     // then what we need is
     // Q <-- S R.
     // It is curious and (to me) unexpected that the actual code is basically
     // the same when transposed.
     CuMatrix<BaseFloat> S(N, N);
     // S <-- lambda I.
     S.AddToDiag(lambda);
     // S += (N-1) R R^T.
     // the following function only updates the lower triangle.
     S.SymAddMat2(1.0 / (N-1), R, kNoTrans, 1.0);
     S.CopyLowerToUpper();
     // invert S, so now S = (\lambda I + (N-1) R R^T)^{-1}.
     if (GetVerboseLevel() >= 5 && Rand() % 20 == 0) {
       CuSpMatrix<BaseFloat> tmp(S, kTakeLower);
       SpMatrix<BaseFloat> S_cpu(tmp);
       S_cpu.PrintEigs("S");
     }
     S.SymInvertPosDef();
     Q.AddMatMat(1.0, S, kNoTrans, R, kNoTrans, 0.0);
   }
 
 #if 0  // Old code before it was optimized for CUDA:
   for (int32 n = 0; n < N; n++) {
     CuSubVector<BaseFloat> r(R, n), q(Q, n);
     BaseFloat gamma = VecVec(r, q), // gamma_n = r_n^T q_n.
                beta = 1 + gamma / (N - 1 - gamma);
     if (!(gamma >= 0.0 && beta > 0.0)) {
       KALDI_ERR << "Bad values encountered in preconditioning: gamma = " << gamma
                 << ", beta = " << beta;
     }
     // Q and P share the same memory.  The result of the
     // scaling below will be output as P.
     q.Scale(beta);
   }
 #else
   CuVector<BaseFloat> gamma(N);
   gamma.AddDiagMatMat(1.0, R, kNoTrans, Q, kTrans, 0.0);
   // at this point, gamma(i) equals the i'th row of R dotted with
   // the i'th row of Q.
   Vector<BaseFloat> cpu_gamma(gamma), cpu_beta(N, kUndefined);
   for (int32 n = 0; n < N; n++) {
     BaseFloat this_gamma = cpu_gamma(n),
         this_beta = 1.0 + this_gamma / (N - 1 - this_gamma);
     if (!(this_gamma >= 0.0 && this_beta > 0.0))
       KALDI_ERR << "Bad values encountered in preconditioning: gamma = "
                 << this_gamma << ", beta = " << this_beta;
     cpu_beta(n) = this_beta;
   }
   CuVector<BaseFloat> beta(cpu_beta);
   P->MulRowsVec(beta);
 #endif
 }

◆ PreconditionDirectionsAlpha()

void PreconditionDirectionsAlpha	(	const CuMatrixBase< BaseFloat > &	R,
		double	alpha,
		CuMatrixBase< BaseFloat > *	P
	)

This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections.

Definition at line 114 of file nnet-precondition.cc.

References KALDI_ASSERT, KALDI_WARN, kaldi::kTrans, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), PreconditionDirections(), and kaldi::TraceMatMat().

                                 {
   KALDI_ASSERT(alpha > 0.0);
   // probably does not really make sense.
   double t = TraceMatMat(R, R, kTrans), floor = 1.0e-20;
   if (t < floor) {
     KALDI_WARN << "Flooring trace from " << t
                << " to " << floor;
     t = floor;
   }
   double lambda = t * alpha / R.NumRows() / R.NumCols();
   // see the extended comment below for an explanation of this.
   if (lambda <= 0.0) {
     // This should never really happen, it would probably indicate a bug
     // in the calling code.
     KALDI_WARN << "Zero or negative lambda in PreconditionDirectionsAlpha.";
     lambda = 1.0e-10;
   }
   PreconditionDirections(R, lambda, P);
 }

◆ PreconditionDirectionsAlphaRescaled()

void PreconditionDirectionsAlphaRescaled	(	const CuMatrixBase< BaseFloat > &	R,
		double	alpha,
		CuMatrixBase< BaseFloat > *	P
	)

This wrapper for PreconditionDirections computes lambda using = /(N D) trace(R^T, R), and calls PreconditionDirections.

It then rescales *P so that its 2-norm is the same as that of R.

Definition at line 138 of file nnet-precondition.cc.

References CuMatrixBase< Real >::CopyFromMat(), KALDI_ASSERT, KALDI_WARN, kaldi::kTrans, CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), PreconditionDirections(), CuMatrixBase< Real >::Scale(), and kaldi::TraceMatMat().

Referenced by AffineComponentPreconditioned::Update(), and BlockAffineComponentPreconditioned::Update().

                                 {
   KALDI_ASSERT(alpha > 0.0); // alpha > 1.0
   // probably does not really make sense.
   double t = TraceMatMat(R, R, kTrans), floor = 1.0e-20;
   if (t == 0.0) {
     P->CopyFromMat(R);
     return;
   }
   if (t < floor) {
     KALDI_WARN << "Flooring trace from " << t
                << " to " << floor;
     t = floor;
   }
   double lambda = t * alpha / R.NumRows() / R.NumCols();
   // see the extended comment below for an explanation of this.
   KALDI_ASSERT(lambda != 0.0);
   PreconditionDirections(R, lambda, P);
   double p_trace = TraceMatMat(*P, *P, kTrans),
       rescale = sqrt(t / p_trace);
   KALDI_ASSERT(p_trace != 0.0);
   P->Scale(rescale);
 }

◆ PrintPriorDiagnostics()

void kaldi::nnet2::PrintPriorDiagnostics	(	const Vector< BaseFloat > &	old_priors,
		const Vector< BaseFloat > &	new_priors
	)

Definition at line 48 of file nnet-adjust-priors.cc.

References VectorBase< Real >::AddVec(), VectorBase< Real >::ApplyAbs(), VectorBase< Real >::Dim(), KALDI_LOG, KlDivergence(), and VectorBase< Real >::Max().

Referenced by main().

                                                                 {
   if (old_priors.Dim() == 0) {
     KALDI_LOG << "Model did not previously have priors attached.";
   } else {
     Vector<BaseFloat> diff_prior(new_priors);
     diff_prior.AddVec(-1.0, old_priors);
     diff_prior.ApplyAbs();
     int32 max_index;
     diff_prior.Max(&max_index);
     KALDI_LOG << "Adjusting priors: largest absolute difference was for "
               << "pdf " << max_index << ", " << old_priors(max_index)
               << " -> " << new_priors(max_index);
     KALDI_LOG << "Adjusting priors: K-L divergence from old to new is "
               << KlDivergence(old_priors, new_priors);
   }
 }

◆ ProcessFile() [1/2]

static void kaldi::nnet2::ProcessFile	(	const MatrixBase< BaseFloat > &	feats,
		const Posterior &	pdf_post,
		const std::string &	utt_id,
		int32	left_context,
		int32	right_context,
		int32	num_frames,
		int32	const_feat_dim,
		int64 *	num_frames_written,
		int64 *	num_egs_written,
		NnetExampleWriter *	example_writer
	)

static

Definition at line 32 of file nnet-get-egs.cc.

References NnetExample::input_frames, rnnlm::j, KALDI_ASSERT, NnetExample::labels, NnetExample::left_context, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), MatrixBase< Real >::Row(), NnetExample::spk_info, and TableWriter< Holder >::Write().

Referenced by main().

                                                            {
   KALDI_ASSERT(feats.NumRows() == static_cast<int32>(pdf_post.size()));
   int32 feat_dim = feats.NumCols();
   KALDI_ASSERT(const_feat_dim < feat_dim);
   KALDI_ASSERT(num_frames > 0);
   int32 basic_feat_dim = feat_dim - const_feat_dim;
 
   for (int32 t = 0; t < feats.NumRows(); t += num_frames) {
     int32 this_num_frames = std::min(num_frames,
                                      feats.NumRows() - t);
 
     int32 tot_frames = left_context + this_num_frames + right_context;
     NnetExample eg;
     Matrix<BaseFloat> input_frames(tot_frames, basic_feat_dim);
     eg.left_context = left_context;
     eg.spk_info.Resize(const_feat_dim);
 
     // Set up "input_frames".
     for (int32 j = -left_context; j < this_num_frames + right_context; j++) {
       int32 t2 = j + t;
       if (t2 < 0) t2 = 0;
       if (t2 >= feats.NumRows()) t2 = feats.NumRows() - 1;
       SubVector<BaseFloat> src(feats.Row(t2), 0, basic_feat_dim),
           dest(input_frames, j + left_context);
       dest.CopyFromVec(src);
       if (const_feat_dim > 0) {
         SubVector<BaseFloat> src(feats.Row(t2), basic_feat_dim, const_feat_dim);
         // set eg.spk_info to the average of the corresponding dimensions of
         // the input, taken over the frames whose features we store in the eg.
         eg.spk_info.AddVec(1.0 / tot_frames, src);
       }
     }
     eg.labels.resize(this_num_frames);
     for (int32 j = 0; j < this_num_frames; j++)
       eg.labels[j] = pdf_post[t + j];
     eg.input_frames = input_frames;  // Copy to CompressedMatrix.
     
     std::ostringstream os;
     os << utt_id << "-" << t;
 
     std::string key = os.str(); // key is <utt_id>-<frame_id>
 
     *num_frames_written += this_num_frames;
     *num_egs_written += 1;
 
     example_writer->Write(key, eg);
   }
 }

◆ ProcessFile() [2/2]

static void kaldi::nnet2::ProcessFile	(	const MatrixBase< BaseFloat > &	feats,
		const Posterior &	pdf_post,
		const std::string &	utt_id,
		const Vector< BaseFloat > &	weights,
		int32	left_context,
		int32	right_context,
		int32	const_feat_dim,
		BaseFloat	keep_proportion,
		BaseFloat	weight_threshold,
		bool	use_frame_selection,
		bool	use_frame_weights,
		int64 *	num_frames_written,
		int64 *	num_frames_skipped,
		NnetExampleWriter *	example_writer
	)

static

Definition at line 45 of file nnet-get-weighted-egs.cc.

References VectorBase< Real >::CopyFromVec(), count, GetCount(), rnnlm::i, NnetExample::input_frames, rnnlm::j, KALDI_ASSERT, NnetExample::labels, NnetExample::left_context, MatrixBase< Real >::NumCols(), MatrixBase< Real >::NumRows(), MatrixBase< Real >::Row(), NnetExample::spk_info, and TableWriter< Holder >::Write().

                                                            {
   KALDI_ASSERT(feats.NumRows() == static_cast<int32>(pdf_post.size()));
   int32 feat_dim = feats.NumCols();
   KALDI_ASSERT(const_feat_dim < feat_dim);
   int32 basic_feat_dim = feat_dim - const_feat_dim;
   NnetExample eg;
   Matrix<BaseFloat> input_frames(left_context + 1 + right_context,
                                  basic_feat_dim);
   eg.left_context = left_context;
   // TODO: modify this code, and this binary itself, to support the --num-frames
   // option to allow multiple frames per eg.
   for (int32 i = 0; i < feats.NumRows(); i++) {
     int32 count = GetCount(keep_proportion); // number of times
     // we'll write this out (1 by default).
     if (count > 0) {
       // Set up "input_frames".
       for (int32 j = -left_context; j <= right_context; j++) {
         int32 j2 = j + i;
         if (j2 < 0) j2 = 0;
         if (j2 >= feats.NumRows()) j2 = feats.NumRows() - 1;
         SubVector<BaseFloat> src(feats, j2), dest(input_frames,
                                                   j + left_context);
         dest.CopyFromVec(src);
       }
       eg.labels.push_back(pdf_post[i]);
       eg.input_frames = input_frames;
       if (const_feat_dim > 0) {
         // we'll normally reach here if we're using online-estimated iVectors.
         SubVector<BaseFloat> const_part(feats.Row(i),
                                         basic_feat_dim, const_feat_dim);
         eg.spk_info.CopyFromVec(const_part);
       }
       if (use_frame_selection) {
         if (weights(i) < weight_threshold) {
           (*num_frames_skipped)++;
           continue;
         }
       }
       std::ostringstream os;
       os << utt_id << "-" << i;
       std::string key = os.str(); // key in the archive is the number of the example
 
       for (int32 c = 0; c < count; c++)
         example_writer->Write(key, eg);
     }
   }
 }

◆ ReplaceLastComponents()

void ReplaceLastComponents	(	const Nnet &	src_nnet,
		int32	num_to_remove,
		Nnet *	dest_nnet
	)

Removes the last "num_to_remove" components and adds the components from "src_nnet".

Definition at line 58 of file nnet-functions.cc.

References Component::Copy(), Nnet::GetComponent(), Nnet::Init(), KALDI_ASSERT, and Nnet::NumComponents().

Referenced by main().

                                             {
   KALDI_ASSERT(num_to_remove >= 0 && num_to_remove <= dest_nnet->NumComponents());
   int32 c_orig = dest_nnet->NumComponents() - num_to_remove;
 
   std::vector<Component*> components;
   for (int32 c = 0; c < c_orig; c++)
     components.push_back(dest_nnet->GetComponent(c).Copy());
   for (int32 c = 0; c < src_nnet.NumComponents(); c++)
     components.push_back(src_nnet.GetComponent(c).Copy());
 
   // Re-initialize "dest_nnet" from the resulting list of components.
   // The Init method will take ownership of the pointers in the vector:
   dest_nnet->Init(&components);
 }

◆ RescaleNnet()

void RescaleNnet	(	const NnetRescaleConfig &	rescale_config,
		const std::vector< NnetExample > &	examples,
		Nnet *	nnet
	)

Definition at line 218 of file rescale-nnet.cc.

References NnetRescaler::Rescale().

Referenced by NnetRescaleConfig::Register().

                              {
   NnetRescaler rescaler(rescale_config, examples, nnet);
   rescaler.Rescale();
 }

◆ SetMaxChange()

void kaldi::nnet2::SetMaxChange	(	BaseFloat	max_change,
		Nnet *	nnet
	)

Definition at line 29 of file nnet-modify-learning-rates.cc.

References Nnet::GetComponent(), Nnet::NumComponents(), and AffineComponentPreconditioned::SetMaxChange().

Referenced by main().

                                                     {
   for (int32 c = 0; c < nnet->NumComponents(); c++) {
     Component *component = &(nnet->GetComponent(c));
     AffineComponentPreconditioned *ac =
         dynamic_cast<AffineComponentPreconditioned*>(component);
     if (ac != NULL)
       ac->SetMaxChange(max_change);
   }
 }

◆ SetPriors()

void kaldi::nnet2::SetPriors	(	const TransitionModel &	tmodel,
		const Vector< double > &	transition_accs,
		double	prior_floor,
		AmNnet *	am_nnet
	)

Definition at line 28 of file nnet-train-transitions.cc.

References VectorBase< Real >::Dim(), KALDI_ASSERT, AmNnet::NumPdfs(), TransitionModel::NumPdfs(), AmNnet::SetPriors(), and TransitionModel::TransitionIdToPdf().

Referenced by main().

                                 {
   KALDI_ASSERT(tmodel.NumPdfs() == am_nnet->NumPdfs());
   Vector<BaseFloat> pdf_counts(tmodel.NumPdfs());
   KALDI_ASSERT(transition_accs(0) == 0.0); // There is
   // no zero transition-id.
   for (int32 tid = 1; tid < transition_accs.Dim(); tid++) {
     int32 pdf = tmodel.TransitionIdToPdf(tid);
     pdf_counts(pdf) += transition_accs(tid);
   }
   BaseFloat sum = pdf_counts.Sum();
   KALDI_ASSERT(sum != 0.0);
   KALDI_ASSERT(prior_floor > 0.0 && prior_floor < 1.0);
   pdf_counts.Scale(1.0 / sum);
   pdf_counts.ApplyFloor(prior_floor);
   pdf_counts.Scale(1.0 / pdf_counts.Sum()); // normalize again.
   am_nnet->SetPriors(pdf_counts);
 }               

◆ ShrinkNnet()

void ShrinkNnet	(	const NnetShrinkConfig &	shrink_config,
		const std::vector< NnetExample > &	validation_set,
		Nnet *	nnet
	)

Definition at line 66 of file shrink-nnet.cc.

References VectorBase< Real >::ApplyExp(), ComputeObjfAndGradient(), OptimizeLbfgs< Real >::DoStep(), LbfgsOptions::first_step_length, OptimizeLbfgs< Real >::GetProposedValue(), OptimizeLbfgs< Real >::GetValue(), rnnlm::i, NnetShrinkConfig::initial_step, KALDI_ASSERT, KALDI_LOG, KALDI_VLOG, LbfgsOptions::m, LbfgsOptions::minimize, NnetShrinkConfig::num_bfgs_iters, Nnet::NumUpdatableComponents(), and Nnet::ScaleComponents().

Referenced by NnetShrinkConfig::Register().

                             {
 
   int32 dim = nnet->NumUpdatableComponents();
   KALDI_ASSERT(dim > 0);
   Vector<double> log_scale(dim), gradient(dim); // will be zero.
   
   // Get initial gradient.
   double objf, initial_objf;
 
 
   LbfgsOptions lbfgs_options;
   lbfgs_options.minimize = false; // We're maximizing.
   lbfgs_options.m = dim; // Store the same number of vectors as the dimension
   // itself, so this is BFGS.
   lbfgs_options.first_step_length = shrink_config.initial_step;
   
   OptimizeLbfgs<double> lbfgs(log_scale,
                               lbfgs_options);
   
   for (int32 i = 0; i < shrink_config.num_bfgs_iters; i++) {
     log_scale.CopyFromVec(lbfgs.GetProposedValue());
     objf = ComputeObjfAndGradient(validation_set, log_scale,
                                   *nnet,
                                   &gradient);
 
     KALDI_VLOG(2) << "log-scale = " << log_scale << ", objf = " << objf
                   << ", gradient = " << gradient;
     if (i == 0) initial_objf = objf;
 
     lbfgs.DoStep(objf, gradient);
   }
 
   log_scale.CopyFromVec(lbfgs.GetValue(&objf));
 
   Vector<BaseFloat> scale(log_scale);
   scale.ApplyExp();
   KALDI_LOG << "Shrinking nnet, validation objf per frame changed from "
             << initial_objf << " to " << objf << ", scale factors per layer are "
             << scale;
   nnet->ScaleComponents(scale);
 }

◆ SolvePackingProblem()

void SolvePackingProblem	(	BaseFloat	max_cost,
		const std::vector< BaseFloat > &	costs,
		std::vector< std::vector< size_t > > *	groups
	)

This function solves the "packing problem" using the "first fit" algorithm.

It groups together the indices 0 through sizes.size() - 1, such that the sum of cost within each group does not exceed max_lcost. [However, if there are single examples that exceed max_cost, it puts them in their own bin]. The algorithm is not particularly efficient– it's more n^2 than n log(n) which it should be.

Definition at line 867 of file nnet-example-functions.cc.

References rnnlm::i, and rnnlm::j.

Referenced by CombineDiscriminativeExamples(), SplitExampleStats::SplitExampleStats(), and UnitTestSolvePackingProblem().

                                                                 {
   groups->clear();
   std::vector<BaseFloat> group_costs;
   for (size_t i = 0; i < costs.size(); i++) {
     bool found_group = false;
     BaseFloat this_cost = costs[i];
     for (size_t j = 0; j < groups->size(); j++) {
       if (group_costs[j] + this_cost <= max_cost) {
         (*groups)[j].push_back(i);
         group_costs[j] += this_cost;
         found_group = true;
         break;
       }
     }
     if (!found_group) { // Put this object in a newly created group.
       groups->resize(groups->size() + 1);
       groups->back().push_back(i);
       group_costs.push_back(this_cost);
     }
   }
 }

◆ SplitDiscriminativeExample()

void SplitDiscriminativeExample	(	const SplitDiscriminativeExampleConfig &	config,
		const TransitionModel &	tmodel,
		const DiscriminativeNnetExample &	eg,
		std::vector< DiscriminativeNnetExample > *	egs_out,
		SplitExampleStats *	stats_out
	)

Split a "discriminative example" into multiple pieces, splitting where the lattice has "pinch points".

Definition at line 764 of file nnet-example-functions.cc.

References DiscriminativeExampleSplitter::Split().

Referenced by main(), and SplitExampleStats::SplitExampleStats().

                                   {
   DiscriminativeExampleSplitter splitter(config, tmodel, eg, egs_out);
   splitter.Split(stats_out);
 }

◆ TotalNnetTrainingWeight()

BaseFloat TotalNnetTrainingWeight ( const std::vector< NnetExample > & egs )

Returns the total weight summed over all the examples...

just a simple utility function.

Definition at line 248 of file nnet-update.cc.

References rnnlm::i, and rnnlm::j.

Referenced by DoBackpropSingleThreaded(), main(), DoBackpropParallelClass::operator()(), and NnetExampleBackgroundReader::ReadExamples().

                                                                      {
   double ans = 0.0;
   for (size_t i = 0; i < egs.size(); i++)
     for (size_t j = 0; j < egs[i].labels.size(); j++) // for each labeled frame
       for (size_t k = 0; k < egs[i].labels[j].size(); k++)
         ans += egs[i].labels[j][k].second;
   return ans;
 }

◆ TrainNnetSimple()

int64 TrainNnetSimple	(	const NnetSimpleTrainerConfig &	config,
		Nnet *	nnet,
		SequentialNnetExampleReader *	reader,
		double *	tot_weight = `NULL`,
		double *	tot_logprob = `NULL`
	)

Train on all the examples it can read from the reader.

This does training in a single thread, but it uses a separate thread to read in the examples and format the input data on the CPU; this saves us time when using GPUs. Returns the number of examples processed. Outputs to tot_weight and tot_logprob_per_frame, if non-NULL, the total weight of the examples (typically equal to the number of examples) and the total logprob objective function.

Definition at line 147 of file train-nnet.cc.

References DoBackprop(), NnetExampleBackgroundReader::GetNextMinibatch(), rnnlm::i, KALDI_ASSERT, KALDI_LOG, KALDI_WARN, NnetSimpleTrainerConfig::minibatch_size, and NnetSimpleTrainerConfig::minibatches_per_phase.

Referenced by main(), and NnetSimpleTrainerConfig::Register().

                                                {
   int64 num_egs_processed = 0;
   double tot_weight = 0.0, tot_logprob = 0.0;
   NnetExampleBackgroundReader background_reader(config.minibatch_size,
                                                 nnet, reader);
   KALDI_ASSERT(config.minibatches_per_phase > 0);
   while (true) {
     // Iterate over phases.  A phase of training is just a certain number of
     // minibatches, and its only significance is that it's the periodicity with
     // which we print diagnostics.
     double tot_weight_this_phase = 0.0, tot_logprob_this_phase = 0.0;
 
     int32 i;
     for (i = 0; i < config.minibatches_per_phase; i++) {
       std::vector<NnetExample> examples;
       Matrix<BaseFloat> examples_formatted;
       double minibatch_total_weight;  // this will normally equal minibatch size.
       if (!background_reader.GetNextMinibatch(&examples, &examples_formatted,
                                               &minibatch_total_weight))
         break;
       tot_logprob_this_phase += DoBackprop(*nnet, examples, &examples_formatted,
                                            nnet, NULL);
       tot_weight_this_phase += minibatch_total_weight;
       num_egs_processed += examples.size();
     }
     if (i != 0) {
       KALDI_LOG << "Training objective function (this phase) is "
                 << (tot_logprob_this_phase / tot_weight_this_phase) << " over "
                 << tot_weight_this_phase << " frames.";
     }
     tot_weight += tot_weight_this_phase;
     tot_logprob += tot_logprob_this_phase;
     if (i != config.minibatches_per_phase) {
       // did not get all the minibatches we wanted because no more input.
       // this is true if and only if we did "break" in the loop over i above.
       break;
     }
   }
   if (tot_weight == 0.0) {
     KALDI_WARN << "No data seen.";
   } else {
     KALDI_LOG << "Did backprop on " << tot_weight
               << " examples, average log-prob per frame is "
               << (tot_logprob / tot_weight);
     KALDI_LOG << "[this line is to be parsed by a script:] log-prob-per-frame="
               << (tot_logprob / tot_weight);
   }
   if (tot_weight_ptr) *tot_weight_ptr = tot_weight;
   if (tot_logprob_ptr) *tot_logprob_ptr = tot_logprob;
   return num_egs_processed;
 }

◆ UnitTestAdditiveNoiseComponent()

void kaldi::nnet2::UnitTestAdditiveNoiseComponent ( )

Definition at line 434 of file nnet-component-test.cc.

References rnnlm::i, AdditiveNoiseComponent::InitFromString(), KALDI_ERR, KALDI_WARN, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                       {
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   int32 num_fail = 0, num_tries = 4;
   for (int32 i = 0; i < num_tries; i++) {
     try {
       int32 input_dim = 10 + Rand() % 50;
       {
         AdditiveNoiseComponent additive_noise_component(input_dim, 0.1);
         UnitTestGenericComponentInternal(additive_noise_component);
       }
       {
         AdditiveNoiseComponent additive_noise_component;
         additive_noise_component.InitFromString("dim=15 stddev=0.2");
         UnitTestGenericComponentInternal(additive_noise_component);
       }
     } catch (...) {
       KALDI_WARN << "Ignoring failure in AdditiveNoiseComponent test";
       num_fail++;
     }
   }
   if (num_fail >= num_tries/2) {
     KALDI_ERR << "Too many test failures.";
   }
 }

◆ UnitTestAffineComponent()

void kaldi::nnet2::UnitTestAffineComponent ( )

Definition at line 337 of file nnet-component-test.cc.

References AffineComponent::Init(), AffineComponent::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

                                {
   BaseFloat learning_rate = 0.01,
       param_stddev = 0.1, bias_stddev = 1.0;
   int32 input_dim = 5 + Rand() % 10, output_dim = 5 + Rand() % 10;
   {
     AffineComponent component;
     if (Rand() % 2 == 0) {
       component.Init(learning_rate, input_dim, output_dim,
                      param_stddev, bias_stddev);
     } else {
       Matrix<BaseFloat> mat(output_dim + 1, input_dim);
       mat.SetRandn();
       mat.Scale(param_stddev);
       WriteKaldiObject(mat, "tmpf", true);
       Sleep(0.5);
       component.Init(learning_rate, "tmpf");
       unlink("tmpf");
     }
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "learning-rate=0.01 input-dim=10 output-dim=15 param-stddev=0.1";
     AffineComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestAffineComponentPreconditioned()

void kaldi::nnet2::UnitTestAffineComponentPreconditioned ( )

Definition at line 478 of file nnet-component-test.cc.

References AffineComponentPreconditioned::Init(), AffineComponentPreconditioned::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

                                              {
   BaseFloat learning_rate = 0.01,
       param_stddev = 0.1, bias_stddev = 1.0, alpha = 0.01,
       max_change = 100.0;
   int32 input_dim = 5 + Rand() % 10, output_dim = 5 + Rand() % 10;
   {
     AffineComponentPreconditioned component;
     if (Rand() % 2 == 0) {
       component.Init(learning_rate, input_dim, output_dim,
                      param_stddev, bias_stddev,
                      alpha, max_change);
     } else {
       Matrix<BaseFloat> mat(output_dim + 1, input_dim);
       mat.SetRandn();
       mat.Scale(param_stddev);
       WriteKaldiObject(mat, "tmpf", true);
       Sleep(0.5);
       component.Init(learning_rate, alpha, max_change, "tmpf");
       unlink("tmpf");
     }
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "learning-rate=0.01 input-dim=16 output-dim=15 param-stddev=0.1 alpha=0.01";
     AffineComponentPreconditioned component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestAffineComponentPreconditionedOnline()

void kaldi::nnet2::UnitTestAffineComponentPreconditionedOnline ( )

Definition at line 509 of file nnet-component-test.cc.

References AffineComponentPreconditionedOnline::Init(), AffineComponentPreconditionedOnline::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

                                                    {
   BaseFloat learning_rate = 0.01,
       param_stddev = 0.1, bias_stddev = 1.0, num_samples_history = 2000.0, alpha = 4.0,
       max_change_per_sample = 0.1, update_period = 1;
   int32 input_dim = 5 + Rand() % 10, output_dim = 5 + Rand() % 10,
       rank_in = 1 + Rand() % 5, rank_out = 1 + Rand() % 5;
   {
     AffineComponentPreconditionedOnline component;
     if (Rand() % 2 == 0) {
       component.Init(learning_rate, input_dim, output_dim,
                      param_stddev, bias_stddev,
                      rank_in, rank_out, update_period,
                      num_samples_history, alpha,
                      max_change_per_sample);
     } else {
       Matrix<BaseFloat> mat(output_dim + 1, input_dim);
       mat.SetRandn();
       mat.Scale(param_stddev);
       WriteKaldiObject(mat, "tmpf", true);
       Sleep(0.5);
       component.Init(learning_rate, rank_in, rank_out,
                      update_period, num_samples_history, alpha,
                      max_change_per_sample, "tmpf");
       unlink("tmpf");
     }
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "learning-rate=0.01 input-dim=16 output-dim=15 param-stddev=0.1 num-samples-history=3000 alpha=2.0 update-period=1 rank-in=5 rank-out=6";
     AffineComponentPreconditionedOnline component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestAmNnet()

void kaldi::nnet2::UnitTestAmNnet ( )

Definition at line 29 of file am-nnet-test.cc.

References VectorBase< Real >::ApplyExp(), kaldi::GenRandContextDependencyLarge(), GenRandomNnet(), kaldi::GetDefaultTopology(), rnnlm::i, KALDI_ASSERT, TransitionModel::NumPdfs(), AmNnet::Read(), VectorBase< Real >::Scale(), AmNnet::SetPriors(), VectorBase< Real >::SetRandn(), VectorBase< Real >::Sum(), and AmNnet::Write().

Referenced by main().

                       {
   std::vector<int32> phones;
   phones.push_back(1);
   for (int32 i = 2; i < 20; i++)
     if (rand() % 2 == 0)
       phones.push_back(i);
   int32 N = 2 + rand() % 2, // context-size N is 2 or 3.
       P = rand() % N;  // Central-phone is random on [0, N)
 
   std::vector<int32> num_pdf_classes;
 
   ContextDependency *ctx_dep =
       GenRandContextDependencyLarge(phones, N, P,
                                     true, &num_pdf_classes);
 
   HmmTopology topo = GetDefaultTopology(phones);
 
   TransitionModel trans_model(*ctx_dep, topo);
 
   delete ctx_dep; // We won't need this further.
   ctx_dep = NULL;
 
   int32 input_dim = 40, output_dim = trans_model.NumPdfs();
   Nnet *nnet = GenRandomNnet(input_dim, output_dim);
 
   AmNnet am_nnet(*nnet);
   delete nnet;
   nnet = NULL;
   Vector<BaseFloat> priors(output_dim);
   priors.SetRandn();
   priors.ApplyExp();
   priors.Scale(1.0 / priors.Sum());
 
   am_nnet.SetPriors(priors);
 
   bool binary = (rand() % 2 == 0);
   std::ostringstream os;
   am_nnet.Write(os, binary);
   AmNnet am_nnet2;
   std::istringstream is(os.str());
   am_nnet2.Read(is, binary);
 
   std::ostringstream os2;
   am_nnet2.Write(os2, binary);
 
   KALDI_ASSERT(os2.str() == os.str());
 }

◆ UnitTestBlockAffineComponent()

void kaldi::nnet2::UnitTestBlockAffineComponent ( )

Definition at line 544 of file nnet-component-test.cc.

References BlockAffineComponent::Init(), BlockAffineComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                     {
   BaseFloat learning_rate = 0.01,
       param_stddev = 0.1, bias_stddev = 0.1;
   int32 num_blocks = 1 + Rand() % 3,
          input_dim = num_blocks * (2 + Rand() % 4),
         output_dim = num_blocks * (2 + Rand() % 4);
 
   {
     BlockAffineComponent component;
     component.Init(learning_rate, input_dim, output_dim,
                    param_stddev, bias_stddev, num_blocks);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "learning-rate=0.01 input-dim=10 output-dim=15 param-stddev=0.1 num-blocks=5";
     BlockAffineComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestBlockAffineComponentPreconditioned()

void kaldi::nnet2::UnitTestBlockAffineComponentPreconditioned ( )

Definition at line 565 of file nnet-component-test.cc.

References BlockAffineComponentPreconditioned::Init(), BlockAffineComponentPreconditioned::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                                   {
   BaseFloat learning_rate = 0.01,
       param_stddev = 0.1, bias_stddev = 1.0, alpha = 3.0;
   int32 num_blocks = 1 + Rand() % 3,
          input_dim = num_blocks * (2 + Rand() % 4),
         output_dim = num_blocks * (2 + Rand() % 4);
 
   {
     BlockAffineComponentPreconditioned component;
     component.Init(learning_rate, input_dim, output_dim,
                    param_stddev, bias_stddev, num_blocks, alpha);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "learning-rate=0.01 input-dim=10 output-dim=15 param-stddev=0.1 num-blocks=5 alpha=3.0";
     BlockAffineComponentPreconditioned component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestConvolutional1dComponent()

void kaldi::nnet2::UnitTestConvolutional1dComponent ( )

Definition at line 365 of file nnet-component-test.cc.

References Convolutional1dComponent::Init(), Convolutional1dComponent::InitFromString(), kaldi::Rand(), MatrixBase< Real >::Scale(), MatrixBase< Real >::SetRandn(), kaldi::Sleep(), UnitTestGenericComponentInternal(), and kaldi::WriteKaldiObject().

Referenced by main().

                                         {
   BaseFloat learning_rate = 0.01,
             param_stddev = 0.1, bias_stddev = 1.0;
   int32 patch_stride = 10, patch_step = 1, patch_dim = 4;
   int32 num_patches = 1 + (patch_stride - patch_dim) / patch_step;
   int32 num_splice = 5 + Rand() % 10, num_filters = 5 + Rand() % 10;
   int32 input_dim = patch_stride * num_splice;
   int32 filter_dim = patch_dim * num_splice;
   int32 output_dim = num_patches * num_filters;
   {
     Convolutional1dComponent component;
     if (Rand() % 2 == 0) {
       component.Init(learning_rate, input_dim, output_dim,
                      patch_dim, patch_step, patch_stride,
                      param_stddev, bias_stddev, true);
     } else {
       Matrix<BaseFloat> mat(num_filters, filter_dim + 1);
       mat.SetRandn();
       mat.Scale(param_stddev);
       WriteKaldiObject(mat, "tmpf", true);
       Sleep(0.5);
       component.Init(learning_rate, patch_dim,
                      patch_step, patch_stride, "tmpf", false);
       unlink("tmpf");
     }
     UnitTestGenericComponentInternal(component);
   }
   {
     // appended-conv is false by default
     const char *str = "learning-rate=0.01 input-dim=100 output-dim=70 param-stddev=0.1 patch-dim=4 patch-step=1 patch-stride=10";
     Convolutional1dComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "learning-rate=0.01 input-dim=100 output-dim=70 param-stddev=0.1 patch-dim=4 patch-step=1 patch-stride=10 appended-conv=true";
     Convolutional1dComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestDctComponent()

void kaldi::nnet2::UnitTestDctComponent ( )

Definition at line 607 of file nnet-component-test.cc.

References DctComponent::Init(), DctComponent::InitFromString(), rnnlm::n, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                             {
   int32 m = 3 + Rand() % 4, n = 3 + Rand() % 4,
   dct_dim = m, dim = m * n;
   bool reorder = (Rand() % 2 == 0);
   {
     DctComponent component;
     component.Init(dim, dct_dim, reorder);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "dim=10 dct-dim=5 reorder=true";
     DctComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=2";
     DctComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=3";
     DctComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "dim=10 dct-dim=5 reorder=true dct-keep-dim=4";
     DctComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestDropoutComponent()

void kaldi::nnet2::UnitTestDropoutComponent ( )

Definition at line 407 of file nnet-component-test.cc.

References rnnlm::i, DropoutComponent::InitFromString(), KALDI_ERR, KALDI_WARN, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                 {
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   int32 num_fail = 0, num_tries = 4;
   for (int32 i = 0; i < num_tries; i++) {
     try {
       int32 input_dim = 10 + Rand() % 50;
       {
         DropoutComponent dropout_component(input_dim, 0.5, 0.3);
         UnitTestGenericComponentInternal(dropout_component);
       }
       {
         DropoutComponent dropout_component;
         dropout_component.InitFromString("dim=15 dropout-proportion=0.6 dropout-scale=0.1");
         UnitTestGenericComponentInternal(dropout_component);
       }
     } catch (...) {
       KALDI_WARN << "Ignoring test failure in UnitTestDropoutComponent().";
       num_fail++;
     }
   }
   if (num_fail >= num_tries/2) {
     KALDI_ERR << "Too many test failures.";
   }
 }

◆ UnitTestFixedAffineComponent()

void kaldi::nnet2::UnitTestFixedAffineComponent ( )

Definition at line 655 of file nnet-component-test.cc.

References FixedAffineComponent::Init(), rnnlm::n, kaldi::Rand(), CuMatrixBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

                                     {
   int32 m = 15 + Rand() % 4, n = 15 + Rand() % 4;
   {
     CuMatrix<BaseFloat> mat(m, n);
     mat.SetRandn();
     FixedAffineComponent component;
     component.Init(mat);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestFixedBiasComponent()

void kaldi::nnet2::UnitTestFixedBiasComponent ( )

Definition at line 677 of file nnet-component-test.cc.

References FixedBiasComponent::Init(), kaldi::Rand(), CuVectorBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

                                   {
   int32 m = 1 + Rand() % 20;
   {
     CuVector<BaseFloat> vec(m);
     vec.SetRandn();
     FixedBiasComponent component;
     component.Init(vec);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestFixedLinearComponent()

void kaldi::nnet2::UnitTestFixedLinearComponent ( )

Definition at line 643 of file nnet-component-test.cc.

References FixedLinearComponent::Init(), rnnlm::n, kaldi::Rand(), CuMatrixBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

                                     {
   int32 m = 1 + Rand() % 4, n = 1 + Rand() % 4;
   {
     CuMatrix<BaseFloat> mat(m, n);
     mat.SetRandn();
     FixedLinearComponent component;
     component.Init(mat);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestFixedScaleComponent()

void kaldi::nnet2::UnitTestFixedScaleComponent ( )

Definition at line 666 of file nnet-component-test.cc.

References FixedScaleComponent::Init(), kaldi::Rand(), CuVectorBase< Real >::SetRandn(), and UnitTestGenericComponentInternal().

Referenced by main().

                                    {
   int32 m = 1 + Rand() % 20;
   {
     CuVector<BaseFloat> vec(m);
     vec.SetRandn();
     FixedScaleComponent component;
     component.Init(vec);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestGenericComponent()

void kaldi::nnet2::UnitTestGenericComponent ( std::string extra_str = "" )

Definition at line 244 of file nnet-component-test.cc.

References kaldi::Rand(), and UnitTestGenericComponentInternal().

                                                         {
   // works if it has an initializer from int,
   // e.g. tanh, sigmoid.
 
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   int32 input_dim = 10 + Rand() % 50;
   {
     T component(input_dim);
     UnitTestGenericComponentInternal(component);
   }
   {
     T component;
     component.InitFromString(static_cast<std::string>("dim=15 ") + extra_str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestGenericComponentInternal() [1/2]

void kaldi::nnet2::UnitTestGenericComponentInternal	(	const Component &	component,
		const ChunkInfo	in_info,
		const ChunkInfo	out_info
	)

Definition at line 28 of file nnet-component-test.cc.

References CuVectorBase< Real >::AddMatVec(), Component::Backprop(), Component::BackpropNeedsInput(), Component::BackpropNeedsOutput(), Component::Copy(), UpdatableComponent::DotProduct(), rnnlm::i, KALDI_ASSERT, KALDI_ERR, KALDI_LOG, KALDI_WARN, kaldi::kNoTrans, kaldi::kTrans, ChunkInfo::NumCols(), ChunkInfo::NumRows(), UpdatableComponent::PerturbParams(), Component::Propagate(), kaldi::Rand(), Component::ReadNew(), RandomComponent::ResetGenerator(), CuMatrixBase< Real >::Row(), CuVectorBase< Real >::SetRandn(), CuMatrixBase< Real >::SetRandn(), Output::Stream(), Input::Stream(), kaldi::TraceMatMat(), and Component::Write().

                                                                  {
 
   CuMatrix<BaseFloat> input(in_info.NumRows(), in_info.NumCols()),
       output(1, out_info.NumRows() * out_info.NumCols());
   input.SetRandn();
   CuVector<BaseFloat> objf_vec(out_info.NumCols()); // objective function is linear function of output.
   objf_vec.SetRandn(); // set to Gaussian noise.
 
   int32 rand_seed = Rand();
 
   RandomComponent *rand_component =
       const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
   if (rand_component != NULL) {
     srand(rand_seed);
     rand_component->ResetGenerator();
   }
   component.Propagate(in_info, out_info, input, &output);
   {
     bool binary = (Rand() % 2 == 0);
     Output ko("tmpf", binary);
     component.Write(ko.Stream(), binary);
   }
   Component *component_copy;
   {
     bool binary_in;
     Input ki("tmpf", &binary_in);
     component_copy = Component::ReadNew(ki.Stream(), binary_in);
   }
   unlink("tmpf");
 
   { // Test backward derivative is correct.
     CuVector<BaseFloat> output_objfs(out_info.NumRows());
     output_objfs.AddMatVec(1.0, output, kNoTrans, objf_vec, 0.0);
     BaseFloat objf = output_objfs.Sum();
 
 
     CuMatrix<BaseFloat> output_deriv(output.NumRows(), output.NumCols());
     for (int32 i = 0; i < output_deriv.NumRows(); i++)
       output_deriv.Row(i).CopyFromVec(objf_vec);
 
     CuMatrix<BaseFloat> input_deriv(input.NumRows(), input.NumCols());
 
 
     CuMatrix<BaseFloat> empty_mat;
     CuMatrix<BaseFloat> &input_ref =
         (component_copy->BackpropNeedsInput() ? input : empty_mat),
         &output_ref =
         (component_copy->BackpropNeedsOutput() ? output : empty_mat);
 
     component_copy->Backprop(in_info, out_info, input_ref, output_ref,
                              output_deriv, NULL, &input_deriv);
 
     int32 num_ok = 0, num_bad = 0, num_tries = 10;
     KALDI_LOG << "Comparing feature gradients " << num_tries << " times.";
     for (int32 i = 0; i < num_tries; i++) {
       CuMatrix<BaseFloat> perturbed_input(input.NumRows(), input.NumCols());
       {
         RandomComponent *rand_component =
             const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
         if (rand_component != NULL) {
           srand(rand_seed);
           rand_component->ResetGenerator();
         }
       }
       perturbed_input.SetRandn();
       perturbed_input.Scale(1.0e-04); // scale by a small amount so it's like a delta.
       BaseFloat predicted_difference = TraceMatMat(perturbed_input,
                                                    input_deriv, kTrans);
       perturbed_input.AddMat(1.0, input); // now it's the input + a delta.
       { // Compute objf with perturbed input and make sure it matches
         // prediction.
         CuMatrix<BaseFloat> perturbed_output(output.NumRows(), output.NumCols());
         {
           RandomComponent *rand_component =
               const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
           if (rand_component != NULL) {
             srand(rand_seed);
             rand_component->ResetGenerator();
           }
         }
         component.Propagate(in_info, out_info, perturbed_input, &perturbed_output);
         CuVector<BaseFloat> perturbed_output_objfs(out_info.NumRows());
         perturbed_output_objfs.AddMatVec(1.0, perturbed_output, kNoTrans,
                                          objf_vec, 0.0);
         BaseFloat perturbed_objf = perturbed_output_objfs.Sum(),
              observed_difference = perturbed_objf - objf;
         KALDI_LOG << "Input gradients: comparing " << predicted_difference
                   << " and " << observed_difference;
         if (fabs(predicted_difference - observed_difference) >
             0.15 * fabs((predicted_difference + observed_difference)/2) &&
             fabs(predicted_difference - observed_difference) > 1.0e-06) {
           KALDI_WARN << "Bad difference!";
           num_bad++;
         } else {
           num_ok++;
         }
       }
     }
     KALDI_LOG << "Succeeded for " << num_ok << " out of " << num_tries
               << " tries.";
     if (num_ok <= num_bad) {
       delete component_copy;
       KALDI_ERR << "Feature-derivative check failed";
     }
   }
 
   UpdatableComponent *ucomponent =
       dynamic_cast<UpdatableComponent*>(component_copy);
 
   if (ucomponent != NULL) { // Test parameter derivative is correct.
 
     int32 num_ok = 0, num_bad = 0, num_tries = 10;
     KALDI_LOG << "Comparing model gradients " << num_tries << " times.";
     for (int32 i = 0; i < num_tries; i++) {
       UpdatableComponent *perturbed_ucomponent =
           dynamic_cast<UpdatableComponent*>(ucomponent->Copy()),
           *gradient_ucomponent =
           dynamic_cast<UpdatableComponent*>(ucomponent->Copy());
       KALDI_ASSERT(perturbed_ucomponent != NULL);
       gradient_ucomponent->SetZero(true); // set params to zero and treat as gradient.
       BaseFloat perturb_stddev = 5.0e-04;
       perturbed_ucomponent->PerturbParams(perturb_stddev);
 
       CuVector<BaseFloat> output_objfs(out_info.NumRows());
       output_objfs.AddMatVec(1.0, output, kNoTrans, objf_vec, 0.0);
       BaseFloat objf = output_objfs.Sum();
 
       CuMatrix<BaseFloat> output_deriv(output.NumRows(), output.NumCols());
       for (int32 i = 0; i < output_deriv.NumRows(); i++)
         output_deriv.Row(i).CopyFromVec(objf_vec);
       CuMatrix<BaseFloat> input_deriv; // (input.NumRows(), input.NumCols());
 
       // This will compute the parameter gradient.
       ucomponent->Backprop(in_info, out_info, input, output, output_deriv,
                            gradient_ucomponent, &input_deriv);
 
       // Now compute the perturbed objf.
       BaseFloat objf_perturbed;
       {
         CuMatrix<BaseFloat> output_perturbed; // (num_egs, output_dim);
         {
           RandomComponent *rand_component =
               const_cast<RandomComponent*>(dynamic_cast<const RandomComponent*>(&component));
           if (rand_component != NULL) {
             srand(rand_seed);
             rand_component->ResetGenerator();
           }
         }
         perturbed_ucomponent->Propagate(in_info, out_info, input, &output_perturbed);
         CuVector<BaseFloat> output_objfs_perturbed(out_info.NumRows());
         output_objfs_perturbed.AddMatVec(1.0, output_perturbed,
                                          kNoTrans, objf_vec, 0.0);
         objf_perturbed = output_objfs_perturbed.Sum();
       }
 
       BaseFloat delta_objf_observed = objf_perturbed - objf,
           delta_objf_predicted = (perturbed_ucomponent->DotProduct(*gradient_ucomponent) -
                                   ucomponent->DotProduct(*gradient_ucomponent));
 
       KALDI_LOG << "Model gradients: comparing " << delta_objf_observed
                 << " and " << delta_objf_predicted;
       if (fabs(delta_objf_predicted - delta_objf_observed) >
           0.05 * (fabs(delta_objf_predicted + delta_objf_observed)/2) &&
           fabs(delta_objf_predicted - delta_objf_observed) > 1.0e-06) {
         KALDI_WARN << "Bad difference!";
         num_bad++;
       } else {
         num_ok++;
       }
       delete perturbed_ucomponent;
       delete gradient_ucomponent;
     }
     if (num_ok < num_bad) {
       delete component_copy;
       KALDI_ERR << "model-derivative check failed";
     }
   }
   delete component_copy; // No longer needed.
 }

◆ UnitTestGenericComponentInternal() [2/2]

void kaldi::nnet2::UnitTestGenericComponentInternal ( const Component & component )

Definition at line 210 of file nnet-component-test.cc.

References Component::Info(), Component::InputDim(), KALDI_LOG, Component::OutputDim(), kaldi::Rand(), and UnitTestGenericComponentInternal().

                                                                   {
   int32 input_dim = component.InputDim(),
       output_dim = component.OutputDim();
 
   KALDI_LOG << component.Info();
   int32 num_egs = 10 + Rand() % 5;
   int32 num_chunks = 1,
         first_offset = 0,
         last_offset = num_egs-1;
 
   ChunkInfo in_info(input_dim, num_chunks, first_offset, last_offset);
   ChunkInfo out_info(output_dim, num_chunks, first_offset, last_offset);
   UnitTestGenericComponentInternal(component, in_info, out_info);
 }

◆ UnitTestMaxoutComponent()

void kaldi::nnet2::UnitTestMaxoutComponent ( )

Definition at line 263 of file nnet-component-test.cc.

References rnnlm::i, MaxoutComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                {
   // works if it has an initializer from int,
   // e.g. tanh, sigmoid.
 
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   for (int32 i = 0; i < 5; i++) {
     int32 output_dim = 10 + Rand() % 20,
         group_size = 1 + Rand() % 10,
         input_dim = output_dim * group_size;
 
     MaxoutComponent component(input_dim, output_dim);
     UnitTestGenericComponentInternal(component);
   }
 
   {
     MaxoutComponent component;
     component.InitFromString("input-dim=15 output-dim=5");
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestMaxpoolingComponent()

void kaldi::nnet2::UnitTestMaxpoolingComponent ( )

Definition at line 310 of file nnet-component-test.cc.

References rnnlm::i, MaxpoolingComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                    {
   // works if it has an initializer from int,
   // e.g. tanh, sigmoid.
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   for (int32 i = 0; i < 5; i++) {
     int32 pool_stride = 5 + Rand() % 10,
           pool_size = 2 + Rand() % 3,
           num_pools = 1 + Rand() % 10;
     int32 output_dim = num_pools * pool_stride;
     int32 num_patches = num_pools * pool_size;
     int32 input_dim = pool_stride * num_patches;
 
     MaxpoolingComponent component(input_dim, output_dim,
                                   pool_size, pool_stride);
     UnitTestGenericComponentInternal(component);
   }
 
   {
     MaxpoolingComponent component;
     component.InitFromString("input-dim=192 output-dim=64 pool-size=3 pool-stride=16");
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestNnet()

void kaldi::nnet2::UnitTestNnet ( )

Definition at line 26 of file nnet-nnet-test.cc.

References GenRandomNnet(), KALDI_ASSERT, Nnet::Read(), and Nnet::Write().

Referenced by main().

                     {
   int32 input_dim = 40, output_dim = 500;
   Nnet *nnet = GenRandomNnet(input_dim, output_dim);
 
   bool binary = (rand() % 2 == 0);
   std::ostringstream os;
   nnet->Write(os, binary);
   Nnet nnet2;
   std::istringstream is(os.str());
   nnet2.Read(is, binary);
 
   std::ostringstream os2;
   nnet2.Write(os2, binary);
 
   KALDI_ASSERT(os2.str() == os.str());
   delete nnet;
 }

◆ UnitTestNnetCompute()

void kaldi::nnet2::UnitTestNnetCompute ( )

Definition at line 29 of file nnet-compute-test.cc.

References kaldi::AssertEqual(), NnetOnlineComputer::Compute(), NnetOnlineComputer::Flush(), GenRandomNnet(), rnnlm::i, Nnet::Info(), KALDI_LOG, Nnet::LeftContext(), NnetComputation(), CuMatrixBase< Real >::NumCols(), CuMatrixBase< Real >::NumRows(), Nnet::RightContext(), and CuMatrixBase< Real >::SetRandn().

Referenced by main().

                            {
   int32 input_dim = 10 + rand() % 40, output_dim = 100 + rand() % 500;
   bool pad_input = (rand() % 2 == 0);
   
   Nnet *nnet = GenRandomNnet(input_dim, output_dim);
   KALDI_LOG << "Left context = " << nnet->LeftContext() << ", right context = "
             << nnet->RightContext() << ", pad-input = " << pad_input;
   KALDI_LOG << "NNet info is " << nnet->Info();
   int32 num_feats = 5 + rand() % 1000;
   CuMatrix<BaseFloat> input(num_feats, input_dim);
   input.SetRandn();
 
   int32 num_output_rows = num_feats -
       (pad_input ? 0 : nnet->LeftContext() + nnet->RightContext());
   if (num_output_rows <= 0)
     return;
   CuMatrix<BaseFloat> output1(num_output_rows, output_dim);
   NnetComputation(*nnet, input, pad_input, &output1);
   CuMatrix<BaseFloat> output2(output1.NumRows(), output1.NumCols());
   int32 cur_input_pos = 0, cur_output_pos = 0;
 
   NnetOnlineComputer computer(*nnet, pad_input);
   while (cur_input_pos <= num_feats) {
     int32 feats_left = num_feats - cur_input_pos;
     CuMatrix<BaseFloat> output_part;
     if (feats_left > 0) {
       int32 chunk_size = std::min<int32>(1 + rand() % 10, feats_left);
       CuSubMatrix<BaseFloat> input_part(input, cur_input_pos, chunk_size,
                                         0, input_dim);
       computer.Compute(input_part, &output_part);
       cur_input_pos += chunk_size;
     } else {
       computer.Flush(&output_part);
       cur_input_pos++; // will terminate the loop.
     }
     if (output_part.NumRows() != 0) {
       output2.Range(cur_output_pos, output_part.NumRows(),
                     0, output_dim).CopyFromMat(output_part);
       cur_output_pos += output_part.NumRows();
     }
   }  
   AssertEqual(output1, output2);
   for (int32 i = 0; i < output1.NumRows(); i++) {
     // just double-check that the frames near the end are right, in case
     // the test above somehow passed despite that.
     if (i < 10 || output1.NumRows() - i < 10) {
       CuSubVector<BaseFloat> vec1(output1, i), vec2(output2, i);
       AssertEqual(vec1, vec2);
     }
   }
   KALDI_LOG << "OK";
   delete nnet;
 }

◆ UnitTestNnetComputeChunked()

void kaldi::nnet2::UnitTestNnetComputeChunked ( )

Definition at line 83 of file nnet-compute-test.cc.

References kaldi::AssertEqual(), GenRandomNnet(), rnnlm::i, Nnet::Info(), KALDI_LOG, Nnet::LeftContext(), NnetComputation(), NnetComputationChunked(), MatrixBase< Real >::NumRows(), Nnet::RightContext(), and CuMatrixBase< Real >::SetRandn().

Referenced by main().

                                   {
   int32 input_dim = 10 + rand() % 40, output_dim = 100 + rand() % 500;
   bool pad_input = true;
   
   Nnet *nnet = GenRandomNnet(input_dim, output_dim);
   int32 num_feats = 100 + rand() % 500;
   int32 chunk_size = num_feats / (2 + rand() % 10);
   CuMatrix<BaseFloat> input(num_feats, input_dim);
   input.SetRandn();
 
   KALDI_LOG << "Left context = " << nnet->LeftContext() 
             << ", right context = " << nnet->RightContext() 
             << ", chunk size = " << chunk_size;
   KALDI_LOG << "NNet info is " << nnet->Info();
 
   int32 num_output_rows = num_feats;
   CuMatrix<BaseFloat> cu_output1(num_output_rows, output_dim);
   CuMatrix<BaseFloat> cu_output2(num_output_rows, output_dim);
   NnetComputation(*nnet, input, pad_input, &cu_output1);
   NnetComputationChunked(*nnet, CuMatrix<BaseFloat>(input), chunk_size, 
                          &cu_output2);
   Matrix<BaseFloat> output1(cu_output1);
   Matrix<BaseFloat> output2(cu_output2);
   AssertEqual(output1, output2);
   for (int32 i = 0; i < output1.NumRows(); i++) {
     // just double-check that the frames near the end are right, in case
     // the test above somehow passed despite that.
     if (i < 10 || output1.NumRows() - i < 10) {
       SubVector<BaseFloat> vec1(output1, i), vec2(output2, i);
       AssertEqual(vec1, vec2);
     }
   }
   KALDI_LOG << "OK";
   delete nnet;
 }

◆ UnitTestNnetDecodable()

void kaldi::nnet2::UnitTestNnetDecodable ( )

Definition at line 31 of file online-nnet2-decodable-test.cc.

References DecodableNnet2OnlineOptions::acoustic_scale, VectorBase< Real >::ApplyExp(), kaldi::ApproxEqual(), kaldi::GenRandContextDependencyLarge(), GenRandomNnet(), kaldi::GetDefaultTopology(), rnnlm::i, KALDI_ASSERT, DecodableNnet2Online::LogLikelihood(), DecodableAmNnet::LogLikelihood(), DecodableNnet2OnlineOptions::max_nnet_batch_size, DecodableNnet2Online::NumFramesReady(), DecodableAmNnet::NumFramesReady(), TransitionModel::NumPdfs(), TransitionModel::NumTransitionIds(), DecodableNnet2OnlineOptions::pad_input, VectorBase< Real >::Scale(), AmNnet::SetPriors(), VectorBase< Real >::SetRandn(), MatrixBase< Real >::SetRandn(), and VectorBase< Real >::Sum().

Referenced by main().

                              {
   std::vector<int32> phones;
   phones.push_back(1);
   for (int32 i = 2; i < 20; i++)
     if (rand() % 2 == 0)
       phones.push_back(i);
   int32 N = 2 + rand() % 2, // context-size N is 2 or 3.
       P = rand() % N;  // Central-phone is random on [0, N)
 
   std::vector<int32> num_pdf_classes;
 
   ContextDependency *ctx_dep =
       GenRandContextDependencyLarge(phones, N, P,
                                     true, &num_pdf_classes);
 
   HmmTopology topo = GetDefaultTopology(phones);
 
   TransitionModel trans_model(*ctx_dep, topo);
 
   delete ctx_dep; // We won't need this further.
   ctx_dep = NULL;
 
   int32 input_dim = 40, output_dim = trans_model.NumPdfs();
   Nnet *nnet = GenRandomNnet(input_dim, output_dim);
 
   AmNnet am_nnet(*nnet);
   delete nnet;
   nnet = NULL;
   Vector<BaseFloat> priors(output_dim);
   priors.SetRandn();
   priors.ApplyExp();
   priors.Scale(1.0 / priors.Sum());
 
   am_nnet.SetPriors(priors);
 
   DecodableNnet2OnlineOptions opts;
   opts.max_nnet_batch_size = 20;
   opts.acoustic_scale = 0.1;
 
   opts.pad_input = (rand() % 2 == 0);
 
   int32 num_input_frames = 400;
   Matrix<BaseFloat> input_feats(num_input_frames, input_dim);
   input_feats.SetRandn();
 
   OnlineMatrixFeature matrix_feature(input_feats);
 
   DecodableNnet2Online online_decodable(am_nnet, trans_model,
                                         opts, &matrix_feature);
 
   DecodableAmNnet offline_decodable(trans_model, am_nnet,
                                     CuMatrix<BaseFloat>(input_feats),
                                     opts.pad_input,
                                     opts.acoustic_scale);
 
   KALDI_ASSERT(online_decodable.NumFramesReady() ==
                offline_decodable.NumFramesReady());
   int32 num_frames = online_decodable.NumFramesReady(),
       num_tids = trans_model.NumTransitionIds();
 
   for (int32 i = 0; i < 50; i++) {
 
     int32 t = rand() % num_frames, tid = 1 + rand() % num_tids;
     BaseFloat l1 = online_decodable.LogLikelihood(t, tid),
         l2 = offline_decodable.LogLikelihood(t, tid);
     KALDI_ASSERT(ApproxEqual(l1, l2));
   }
 }

◆ UnitTestParsing()

void kaldi::nnet2::UnitTestParsing ( )

Definition at line 690 of file nnet-component-test.cc.

References rnnlm::i, KALDI_ASSERT, and ParseFromString().

Referenced by main().

                        {
   int32 i;
   BaseFloat f;
   bool b;
   std::vector<int32> v;
   std::string s = "x=y";
   KALDI_ASSERT(ParseFromString("foo", &s, &i) == false
                && s == "x=y");
   KALDI_ASSERT(ParseFromString("foo", &s, &f) == false
                && s == "x=y");
   KALDI_ASSERT(ParseFromString("foo", &s, &v) == false
                && s == "x=y");
   KALDI_ASSERT(ParseFromString("foo", &s, &b) == false
                && s == "x=y");
   {
     std::string s = "x=1";
     KALDI_ASSERT(ParseFromString("x", &s, &i) == true
                  && i == 1 && s == "");
     s = "a=b x=1";
     KALDI_ASSERT(ParseFromString("x", &s, &i) == true
                  && i == 1 && s == "a=b");
   }
   {
     std::string s = "foo=false";
     KALDI_ASSERT(ParseFromString("foo", &s, &b) == true
                  && b == false && s == "");
     s = "x=y foo=true a=b";
     KALDI_ASSERT(ParseFromString("foo", &s, &b) == true
                  && b == true && s == "x=y a=b");
   }
 
   {
     std::string s = "foobar x=1";
     KALDI_ASSERT(ParseFromString("x", &s, &f) == true
                  && f == 1.0 && s == "foobar");
     s = "a=b x=1 bxy";
     KALDI_ASSERT(ParseFromString("x", &s, &f) == true
                  && f == 1.0 && s == "a=b bxy");
   }
   {
     std::string s = "x=1:2:3";
     KALDI_ASSERT(ParseFromString("x", &s, &v) == true
                  && v.size() == 3 && v[0] == 1 && v[1] == 2 && v[2] == 3
                  && s == "");
     s = "a=b x=1:2:3 c=d";
     KALDI_ASSERT(ParseFromString("x", &s, &v) == true
                  && f == 1.0 && s == "a=b c=d");
   }
 
 }

◆ UnitTestPnormComponent()

void kaldi::nnet2::UnitTestPnormComponent ( )

Definition at line 286 of file nnet-component-test.cc.

References rnnlm::i, KALDI_ERR, KALDI_WARN, kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                               {
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   int32 num_fail = 0, num_tries = 4;
   for (int32 i = 0; i < num_tries; i++) {
     try {
       int32 output_dim = 10 + Rand() % 20,
           group_size = 1 + Rand() % 10,
           input_dim = output_dim * group_size;
       BaseFloat p = 1.0 + 0.1 * (Rand() % 20);
 
       PnormComponent component(input_dim, output_dim, p);
       UnitTestGenericComponentInternal(component);
     } catch (...) {
       KALDI_WARN << "Ignoring test failure in UnitTestPnormComponent().";
       num_fail++;
     }
   }
   if (num_fail >= num_tries/2) {
     KALDI_ERR << "Too many test failures.";
   }
 }

◆ UnitTestPreconditionDirections()

void kaldi::nnet2::UnitTestPreconditionDirections ( )

Definition at line 26 of file nnet-precondition-test.cc.

References CuSpMatrix< Real >::AddMat2(), CuVectorBase< Real >::AddSpVec(), CuSpMatrix< Real >::AddVec2(), CuVectorBase< Real >::ApproxEqual(), CuSpMatrix< Real >::Invert(), KALDI_ASSERT, kaldi::kTrans, rnnlm::n, PreconditionDirections(), kaldi::Rand(), CuPackedMatrix< Real >::ScaleDiag(), CuMatrixBase< Real >::SetRandn(), and CuPackedMatrix< Real >::SetUnit().

Referenced by main().

                                       {
   MatrixIndexT N = 2 + Rand() % 30,
                D = 1 + Rand() % 20;
   BaseFloat lambda = 0.1;
   CuMatrix<BaseFloat> R(N, D), P(N, D);
   R.SetRandn();
   P.SetRandn(); // contents should be overwritten.
 
   PreconditionDirections(R, lambda, &P);
   // The rest of this function will do the computation the function is doing in
   // a different, less efficient way and compare with the function call.
   
   CuSpMatrix<BaseFloat> G(D);
   G.SetUnit();
   G.ScaleDiag(lambda);
   // G += R^T R.
   G.AddMat2(1.0/(N-1), R, kTrans, 1.0);
   
   for (int32 n = 0; n < N; n++) {
     CuSubVector<BaseFloat> rn(R, n);
     CuSpMatrix<BaseFloat> Gn(G);
     Gn.AddVec2(-1.0/(N-1), rn); // subtract the
     // outer product of "this" vector.
     Gn.Invert();
     CuSubVector<BaseFloat> pn(P, n);
     CuVector<BaseFloat> pn_compare(D);
     pn_compare.AddSpVec(1.0, Gn, rn, 0.0);
     KALDI_ASSERT(pn.ApproxEqual(pn_compare, 0.1));
   }
 }

◆ UnitTestPreconditionDirectionsOnline()

void kaldi::nnet2::UnitTestPreconditionDirectionsOnline ( )

Definition at line 262 of file nnet-precondition-online-test.cc.

References CuVectorBase< Real >::AddDiagMatMat(), MatrixBase< Real >::AddVecVec(), kaldi::AssertEqual(), KALDI_ASSERT, kaldi::kNoTrans, kaldi::kTrans, CuMatrixBase< Real >::NumRows(), OnlinePreconditionerSimple::PreconditionDirections(), OnlinePreconditioner::PreconditionDirections(), kaldi::Rand(), kaldi::RandInt(), VectorBase< Real >::Scale(), MatrixBase< Real >::Set(), VectorBase< Real >::SetRandn(), MatrixBase< Real >::SetRandn(), OnlinePreconditionerSimple::SetRank(), OnlinePreconditioner::SetRank(), kaldi::TraceMatMat(), and OnlinePreconditioner::TurnOnDebug().

Referenced by main().

                                             {
   MatrixIndexT R = 1 + Rand() % 30,  // rank of correction
       N = (2 * R) + Rand() % 30,  // batch size
       D = R + 1 + Rand() % 20; // problem dimension.  Must be > R.
 
   // Test sometimes with features that are all-zero or all-one; this will
   // help to make sure low-rank or zero input doesn't crash the code.
   bool zero = false;
   bool one = false;
   if (Rand() % 3 == 0) zero = true;
   //else if (Rand() % 2 == 0) one = true;
 
   CuVector<BaseFloat> row_prod1(N), row_prod2(N);
   BaseFloat gamma1, gamma2;
   BaseFloat big_eig_factor = RandInt(1, 20);
   big_eig_factor = big_eig_factor * big_eig_factor;
   Vector<BaseFloat> big_eig_vector(D);
   big_eig_vector.SetRandn();
   big_eig_vector.Scale(big_eig_factor);
 
   OnlinePreconditionerSimple preconditioner1;
   OnlinePreconditioner preconditioner2;
   preconditioner1.SetRank(R);
   preconditioner2.SetRank(R);
   preconditioner2.TurnOnDebug();
 
   int32 num_iters = 100;
   for (int32 iter = 0; iter < num_iters; iter++) {
     Matrix<BaseFloat> M_cpu(N, D);
     if (one) M_cpu.Set(1.0);
     else if (!zero) {
       M_cpu.SetRandn();
       Vector<BaseFloat> rand_vec(N);
       rand_vec.SetRandn();
       M_cpu.AddVecVec(1.0, rand_vec, big_eig_vector);
     }
     CuMatrix<BaseFloat> M(M_cpu);
 
     CuMatrix<BaseFloat> Mcopy1(M), Mcopy2(M);
 
     preconditioner1.PreconditionDirections(&Mcopy1, &row_prod1, &gamma1);
 
     preconditioner2.PreconditionDirections(&Mcopy2, &row_prod2, &gamma2);
 
     BaseFloat trace1 = TraceMatMat(M, M, kTrans),
         trace2 = TraceMatMat(Mcopy1, Mcopy1, kTrans);
     AssertEqual(trace1, trace2 * gamma2 * gamma2, 1.0e-02);
 
     AssertEqual(Mcopy1, Mcopy2);
     AssertEqual<BaseFloat>(row_prod1, row_prod2, 1.0e-02);
     AssertEqual(gamma1, gamma2, 1.0e-02);
 
     // make sure positive definite
     CuVector<BaseFloat> inner_prods(M.NumRows());
     inner_prods.AddDiagMatMat(1.0, M, kNoTrans, Mcopy1, kTrans, 0.0);
     KALDI_ASSERT(inner_prods.Min() >= 0.0);
   }
   return;
 }

◆ UnitTestScaleComponent()

void kaldi::nnet2::UnitTestScaleComponent ( )

Definition at line 461 of file nnet-component-test.cc.

References ScaleComponent::Init(), ScaleComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                               {
   int32 dim = 1 + Rand() % 10;
   BaseFloat scale = 0.1 + Rand() % 3;
   {
     ScaleComponent component;
     if (Rand() % 2 == 0) {
       component.Init(dim, scale);
     } else {
       std::ostringstream str;
       str << "dim=" << dim << " scale=" << scale;
       component.InitFromString(str.str());
     }
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UnitTestSigmoidComponent()

void kaldi::nnet2::UnitTestSigmoidComponent ( )

Definition at line 227 of file nnet-component-test.cc.

References NonlinearComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                 {
   // We're testing that the gradients are computed correctly:
   // the input gradients and the model gradients.
 
   int32 input_dim = 10 + Rand() % 50;
   {
     SigmoidComponent sigmoid_component(input_dim);
     UnitTestGenericComponentInternal(sigmoid_component);
   }
   {
     SigmoidComponent sigmoid_component;
     sigmoid_component.InitFromString("dim=15");
     UnitTestGenericComponentInternal(sigmoid_component);
   }
 }

◆ UnitTestSolvePackingProblem()

void kaldi::nnet2::UnitTestSolvePackingProblem ( )

Definition at line 29 of file nnet-example-functions-test.cc.

References rnnlm::i, rnnlm::j, KALDI_ASSERT, kaldi::Rand(), SolvePackingProblem(), and kaldi::SortAndUniq().

Referenced by main().

                                    {
   size_t size = Rand() % 20;
   std::vector<BaseFloat> item_costs;
   for (size_t i = 0; i < size; i++) {
     item_costs.push_back(0.5 * (Rand() % 15));
   }
   BaseFloat max_cost = 0.66 + Rand() % 5;
 
   std::vector<std::vector<size_t> > groups;
   SolvePackingProblem(max_cost, item_costs, &groups);
   
   std::vector<size_t> all_indices;
   for (size_t i = 0; i < groups.size(); i++) {
     BaseFloat this_group_cost = 0.0;
     for (size_t j = 0; j < groups[i].size(); j++) {
       size_t index = groups[i][j];
       all_indices.push_back(index);
       this_group_cost += item_costs[index];
     }
     KALDI_ASSERT(!groups[i].empty());
     KALDI_ASSERT(groups[i].size() == 1 || this_group_cost <= max_cost);
   }
   SortAndUniq(&all_indices);
   KALDI_ASSERT(all_indices.size() == size);
   if (!all_indices.empty())
     KALDI_ASSERT(all_indices.back() + 1 == size);
 }

◆ UnitTestSpliceComponent()

void kaldi::nnet2::UnitTestSpliceComponent ( )

Definition at line 741 of file nnet-component-test.cc.

References rnnlm::i, SpliceComponent::Init(), KALDI_LOG, kaldi::RandInt(), and UnitTestGenericComponentInternal().

Referenced by main().

                                {
   int32 feat_dim = RandInt(1, 20),
       const_dim =  RandInt(0, 10),
       left_context = RandInt(-5, 0),
       right_context = RandInt(0, 5),
       num_chunks = RandInt(1, 20);
         // multiple chunks are required as splice component
         // has separate index computation logic for more than one chunks
   KALDI_LOG << " Feat_dim :" << feat_dim << " const_dim: " << const_dim  ;
   std::vector<bool> contiguous(2);
   contiguous[0] = true;
   contiguous[1] = false;
   for (int32 i = 0; i < contiguous.size(); i++) {
     std::vector<int32> splice_indexes;
     if (contiguous[i]) {
       // create contiguous set of splice indexes in the range
       // (-left_context, right_context)
       KALDI_LOG << "Testing contiguous splice component";
       splice_indexes.reserve(right_context - left_context + 1);
       for (int32 i = left_context; i <= right_context; i++)
         splice_indexes.push_back(i);
     } else  {
       // generate random splice indexes in range (-left_context, right_context)
       KALDI_LOG << "Testing non-contiguous splice component";
       int32 num_left_splice_indexes = RandInt(0, -left_context) + 1;
       int32 num_right_splice_indexes = RandInt(0, right_context);
       splice_indexes.reserve(num_left_splice_indexes + num_right_splice_indexes);
       while (splice_indexes.size() < num_left_splice_indexes)  {
         int32 new_index = RandInt(left_context, 0);
         // check if the index already exists in the vector
         if (std::find(splice_indexes.begin(), splice_indexes.end(), new_index)
             == splice_indexes.end())  {
           splice_indexes.push_back(new_index);
         }
       }
       while (splice_indexes.size() < num_left_splice_indexes + num_right_splice_indexes)  {
         int32 new_index = RandInt(0, right_context);
         // check if the index already exists in the vector
         if (std::find(splice_indexes.begin(), splice_indexes.end(), new_index)
             == splice_indexes.end())  {
           splice_indexes.push_back(new_index);
         }
       }
       sort(splice_indexes.begin(), splice_indexes.end());
       if (splice_indexes.back() < 0) // will fail assertion in init of component
         splice_indexes.push_back(0);
     }
     std::vector<int32> input_offsets;
     for (int32 i = 0; i < splice_indexes.size(); i++) {
       input_offsets.push_back(splice_indexes[i] - splice_indexes.front());
       KALDI_LOG << i << " : " << splice_indexes[i] << " : " << input_offsets[i] ;
     }
     int32 output_offset = -splice_indexes.front();
     SpliceComponent *component = new SpliceComponent();
     component->Init(feat_dim + const_dim, splice_indexes, const_dim);
     ChunkInfo in_info = ChunkInfo(feat_dim + const_dim, num_chunks,
                                   input_offsets),
               out_info = ChunkInfo(feat_dim * splice_indexes.size() + const_dim,
                                    num_chunks, output_offset, output_offset);
     UnitTestGenericComponentInternal(*component, in_info, out_info);
     delete component;
   }
 }

◆ UnitTestSumGroupComponent()

void kaldi::nnet2::UnitTestSumGroupComponent ( )

Definition at line 587 of file nnet-component-test.cc.

References rnnlm::i, SumGroupComponent::Init(), SumGroupComponent::InitFromString(), kaldi::Rand(), and UnitTestGenericComponentInternal().

Referenced by main().

                                  {
   std::vector<int32> sizes;
   int32 num_sizes = 1 + Rand() % 5;
   for (int32 i = 0; i < num_sizes; i++)
     sizes.push_back(1 + Rand() % 5);
 
   {
     SumGroupComponent component;
     component.Init(sizes);
     UnitTestGenericComponentInternal(component);
   }
   {
     const char *str = "sizes=3:4:5";
     SumGroupComponent component;
     component.InitFromString(str);
     UnitTestGenericComponentInternal(component);
   }
 }

◆ UpdateHash()

void UpdateHash	(	const TransitionModel &	tmodel,
		const DiscriminativeNnetExample &	eg,
		std::string	criterion,
		bool	drop_frames,
		bool	one_silence_class,
		Matrix< double > *	hash,
		double *	num_weight,
		double *	den_weight,
		double *	tot_t
	)

This function is used in code that tests the functionality that we provide here, about splitting and excising nnet examples.

It adds to a "hash function" that is a function of a set of examples; the hash function is of dimension (number of pdf-ids x features dimension). The hash function consists of the (denominator - numerator) posteriors over pdf-ids, times the average over the context-window (left-context on the left, right-context on the right), of the features. This is useful because the various manipulations we do are supposed to preserve this, and if there is a bug it will most likely cause the hash function to change.

This function will resize the matrix if it is empty.

Any acoustic scaling of the lattice should be done before you call this function.

'criterion' should be 'mmi', 'mpfe', or 'smbr'.

You should set drop_frames to true if you are doing MMI with drop-frames == true. Then it will not compute the hash for frames where the numerator pdf-id is not in the denominator lattice.

You can set one_silence_class to true for a newer optional behavior that will reduce insertions in the trained model (or false for the traditional behavior).

The function will also accumulate the total numerator and denominator weights used as num_weight and den_weight, for an additional diagnostic, and the total number of frames, as tot_t.

Definition at line 786 of file nnet-example-functions.cc.

References VectorBase< Real >::AddRowSumMat(), ExampleToPdfPost(), rnnlm::i, DiscriminativeNnetExample::input_frames, KALDI_ASSERT, DiscriminativeNnetExample::left_context, DiscriminativeNnetExample::num_ali, MatrixBase< Real >::NumCols(), TransitionModel::NumPdfs(), MatrixBase< Real >::NumRows(), Matrix< Real >::Resize(), and MatrixBase< Real >::Row().

Referenced by main(), and SplitExampleStats::SplitExampleStats().

                    {
   int32 feat_dim = eg.input_frames.NumCols(),
       left_context = eg.left_context,
       num_frames = eg.num_ali.size(),
       right_context = eg.input_frames.NumRows() - num_frames - left_context,
       context_width = left_context + 1 + right_context;
   *tot_t += num_frames;
   KALDI_ASSERT(right_context >= 0);
   KALDI_ASSERT(hash != NULL);
   if (hash->NumRows() == 0) {
     hash->Resize(tmodel.NumPdfs(), feat_dim);
   } else {
     KALDI_ASSERT(hash->NumRows() == tmodel.NumPdfs() &&
                  hash->NumCols() == feat_dim);
   }
 
   Posterior post;
   std::vector<int32> silence_phones; // we don't let the user specify this
                                      // because it's not necessary for testing
                                      // purposes -> leave it empty
   ExampleToPdfPost(tmodel, silence_phones, criterion, drop_frames,
                    one_silence_class, eg, &post);
 
   Vector<BaseFloat> avg_feat(feat_dim);
   
   for (int32 t = 0; t < num_frames; t++) {
     SubMatrix<BaseFloat> context_window(eg.input_frames,
                                         t, context_width,
                                         0, feat_dim);
     // set avg_feat to average over the context-window for this frame.
     avg_feat.AddRowSumMat(1.0 / context_width, context_window, 0.0);
     Vector<double> avg_feat_dbl(avg_feat);
     for (size_t i = 0; i < post[t].size(); i++) {
       int32 pdf_id = post[t][i].first;
       BaseFloat weight = post[t][i].second;
       hash->Row(pdf_id).AddVec(weight, avg_feat_dbl);
       if (weight > 0.0) *num_weight += weight;
       else *den_weight += -weight;
     }
   }
 }

◆ WidenNnet()

void WidenNnet	(	const NnetWidenConfig &	widen_config,
		Nnet *	nnet
	)

This function widens a neural network by increasing the hidden-layer dimensions to the target.

Definition at line 62 of file widen-nnet.cc.

References NnetWidenConfig::bias_stddev, Nnet::Check(), Nnet::GetComponent(), NnetWidenConfig::hidden_layer_dim, AffineComponent::InputDim(), KALDI_LOG, Nnet::NumComponents(), AffineComponent::OutputDim(), NnetWidenConfig::param_stddev_factor, and AffineComponent::Widen().

Referenced by main(), and NnetWidenConfig::Register().

                            {
 
   int32 C = nnet->NumComponents();
   int32 num_widened = 0;
 
   for (int32 c = 0; c < C - 3; c++) {
     AffineComponent *c1 = dynamic_cast<AffineComponent*>(&(nnet->GetComponent(c)));
     if (c1 == NULL) continue;
     std::vector<NonlinearComponent*> c2; // normally just one element, but allow two right now.
     c2.push_back(dynamic_cast<NonlinearComponent*>(&(nnet->GetComponent(c+1))));
     if (c2.back() == NULL) continue;
     c2.push_back(dynamic_cast<NonlinearComponent*>(&(nnet->GetComponent(c+2))));
     AffineComponent *c3;
     if (c2.back() == NULL) {
       c2.pop_back();
       c3 = dynamic_cast<AffineComponent*>(&(nnet->GetComponent(c+2)));
     } else {
       if (c + 3 >= C) continue;
       c3 = dynamic_cast<AffineComponent*>(&(nnet->GetComponent(c+3)));
     }
     if (c3 == NULL) continue;
     BaseFloat param_stddev = widen_config.param_stddev_factor /
         sqrt(1.0 * c1->InputDim());
     KALDI_LOG << "Widening component " << c << " from "
               << c1->OutputDim() << " to " << widen_config.hidden_layer_dim;
     
     c1->Widen(widen_config.hidden_layer_dim,
               param_stddev, widen_config.bias_stddev,
               c2, c3);
     num_widened++;
   }
   nnet->Check();
   KALDI_LOG << "Widened " << num_widened << " components.";
 }  

Variable Documentation

◆ nnet_example_warned_left

bool nnet_example_warned_left = false

static

Definition at line 156 of file nnet-example.cc.

◆ nnet_example_warned_right

bool nnet_example_warned_right = false

static

Definition at line 156 of file nnet-example.cc.

Referenced by NnetExample::NnetExample().

Typedefs
typedef TableWriter< KaldiObjectHolder< NnetExample > >	NnetExampleWriter

typedef SequentialTableReader< KaldiObjectHolder< NnetExample > >	SequentialNnetExampleReader

typedef RandomAccessTableReader< KaldiObjectHolder< NnetExample > >	RandomAccessNnetExampleReader

typedef TableWriter< KaldiObjectHolder< DiscriminativeNnetExample > >	DiscriminativeNnetExampleWriter

typedef SequentialTableReader< KaldiObjectHolder< DiscriminativeNnetExample > >	SequentialDiscriminativeNnetExampleReader

typedef RandomAccessTableReader< KaldiObjectHolder< DiscriminativeNnetExample > >	RandomAccessDiscriminativeNnetExampleReader

Classes

Typedefs

Functions

Variables

Typedef Documentation

◆ DiscriminativeNnetExampleWriter

◆ NnetExampleWriter

◆ RandomAccessDiscriminativeNnetExampleReader

◆ RandomAccessNnetExampleReader

◆ SequentialDiscriminativeNnetExampleReader

◆ SequentialNnetExampleReader

Function Documentation

◆ AddDirection()

◆ AppendDiscriminativeExamples()

◆ AverageConstPart()

◆ BasicDebugTestForSpliceMax()

◆ CombineDiscriminativeExamples()

◆ CombineNnets() [1/2]

◆ CombineNnets() [2/2]

◆ CombineNnetsA()

◆ CombineNnetsFast()

◆ ComputeNnetGradient()

◆ ComputeNnetObjf() [1/2]

◆ ComputeNnetObjf() [2/2]

◆ ComputeNnetObjfParallel()

◆ ComputeObjfAndGradient() [1/3]

◆ ComputeObjfAndGradient() [2/3]

◆ ComputeObjfAndGradient() [3/3]

◆ DoBackprop() [1/2]

◆ DoBackprop() [2/2]

◆ DoBackpropParallel() [1/2]

◆ DoBackpropParallel() [2/2]

◆ DoBackpropSingleThreaded()

◆ ExampleToPdfPost()

◆ ExciseDiscriminativeExample()

◆ ExpectOneOrTwoTokens()

◆ FixNnet()

◆ FormatNnetInput()

◆ GenRandomNnet()

◆ GetCount()

◆ GetInitialModel()

◆ GetInitialScaleParams()

◆ GetNnetStats()

◆ GetUpdateDirection()

◆ GiveNnetCorrectTopology()

◆ HasSimpleLabels()

◆ IndexOfSoftmaxLayer()

◆ InsertComponents()

◆ KlDivergence()

◆ LatticeToDiscriminativeExample()

◆ LimitRankParallel()

◆ MakePair()

◆ MixupNnet()

◆ NnetComputation()

◆ NnetComputationChunked()

◆ NnetDiscriminativeUpdate()

◆ NnetDiscriminativeUpdateParallel()

◆ NnetGradientComputation() [1/2]

◆ NnetGradientComputation() [2/2]

◆ ParseFromString() [1/5]

◆ ParseFromString() [2/5]

◆ ParseFromString() [3/5]

◆ ParseFromString() [4/5]

◆ ParseFromString() [5/5]

◆ PreconditionDirections()

◆ PreconditionDirectionsAlpha()

◆ PreconditionDirectionsAlphaRescaled()

◆ PrintPriorDiagnostics()

◆ ProcessFile() [1/2]

◆ ProcessFile() [2/2]

◆ ReplaceLastComponents()

◆ RescaleNnet()

◆ SetMaxChange()

◆ SetPriors()

◆ ShrinkNnet()

◆ SolvePackingProblem()

◆ SplitDiscriminativeExample()

◆ TotalNnetTrainingWeight()

◆ TrainNnetSimple()

◆ UnitTestAdditiveNoiseComponent()