OnlineFeaturePipeline is a class that's responsible for putting together the various stages of the feature-processing pipeline, in an online setting. More...

#include <online-feature-pipeline.h>

Inheritance diagram for OnlineFeaturePipeline:

[legend]

Collaboration diagram for OnlineFeaturePipeline:

[legend]

Public Member Functions
	OnlineFeaturePipeline (const OnlineFeaturePipelineConfig &cfg)

virtual int32	Dim () const
	Member functions from OnlineFeatureInterface: More...

virtual bool	IsLastFrame (int32 frame) const
	Returns true if this is the last frame. More...

virtual int32	NumFramesReady () const
	returns the feature dimension. More...

virtual void	GetFrame (int32 frame, VectorBase< BaseFloat > *feat)
	Gets the feature vector for this frame. More...

void	GetAsMatrix (Matrix< BaseFloat > *feats)

void	FreezeCmvn ()

void	SetCmvnState (const OnlineCmvnState &cmvn_state)
	Set the CMVN state to a particular value (will generally be called after Copy(). More...

void	GetCmvnState (OnlineCmvnState *cmvn_state)

void	AcceptWaveform (BaseFloat sampling_rate, const VectorBase< BaseFloat > &waveform)
	Accept more data to process (won't actually process it, will just copy it). More...

BaseFloat	FrameShiftInSeconds () const

void	InputFinished ()

void	SetTransform (const MatrixBase< BaseFloat > &transform)

bool	HaveFmllrTransform ()

OnlineFeaturePipeline *	New () const
	returns a newly initialized copy of *this– this does not duplicate all the internal state or the speaker-adaptation state, but gives you a freshly initialized version of this object, as if you had initialized it using the constructor that takes the config file. More...

virtual	~OnlineFeaturePipeline ()

Public Member Functions inherited from OnlineFeatureInterface
virtual void	GetFrames (const std::vector< int32 > &frames, MatrixBase< BaseFloat > *feats)
	This is like GetFrame() but for a collection of frames. More...

virtual	~OnlineFeatureInterface ()
	Virtual destructor. More...

Private Member Functions
	OnlineFeaturePipeline (const OnlineFeaturePipelineConfig &cfg, const Matrix< BaseFloat > &lda_mat, const Matrix< BaseFloat > &global_cmvn_stats)
	The following constructor is used internally in the New() function; it has the same effect as initializing from just "cfg", but avoids re-reading the LDA transform from disk. More...

void	Init ()
	Init() is to be called from the constructor; it assumes the pointer members are all uninitialized but config_ and lda_mat_ are initialized. More...

OnlineFeatureInterface *	UnadaptedFeature () const
	returns lda_ if it exists, else splice_or_delta_, else cmvn_. More...

OnlineFeatureInterface *	AdaptedFeature () const
	returns adapted feature if fmllr_ exists, else UnadaptedFeature(). More...

Detailed Description

OnlineFeaturePipeline is a class that's responsible for putting together the various stages of the feature-processing pipeline, in an online setting.

This does not attempt to be fully generic, we just try to handle the common case. Since the online-decoding code needs to "know about" things like CMN and fMLLR in order to do adaptation, it's hard to make this completely generic.

Definition at line 157 of file online-feature-pipeline.h.

Constructor & Destructor Documentation

◆ OnlineFeaturePipeline() [1/2]

OnlineFeaturePipeline ( const OnlineFeaturePipelineConfig & cfg )

explicit

Definition at line 113 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::global_cmvn_stats_, OnlineFeaturePipelineConfig::global_cmvn_stats_rxfilename, OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::lda_mat_, OnlineFeaturePipelineConfig::lda_rxfilename, and kaldi::ReadKaldiObject().

Referenced by OnlineFeaturePipeline::New().

                                               :
     config_(config) {
   if (config.lda_rxfilename != "")
     ReadKaldiObject(config.lda_rxfilename, &lda_mat_);
   if (config.global_cmvn_stats_rxfilename != "")
     ReadKaldiObject(config.global_cmvn_stats_rxfilename,
                     &global_cmvn_stats_);
   Init();
 }

◆ ~OnlineFeaturePipeline()

~OnlineFeaturePipeline ( )

virtual

Definition at line 256 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::base_feature_, OnlineFeaturePipeline::cmvn_, OnlineFeaturePipeline::feature_, OnlineFeaturePipeline::fmllr_, OnlineFeaturePipeline::lda_, OnlineFeaturePipeline::pitch_, OnlineFeaturePipeline::pitch_feature_, and OnlineFeaturePipeline::splice_or_delta_.

                                               {
   // Note: the delete command only deletes pointers that are non-NULL.  Not all
   // of the pointers below will be non-NULL.
   delete fmllr_;
   delete lda_;
   delete splice_or_delta_;
   // Guard against double deleting the cmvn_ ptr
   if (pitch_feature_) {
     delete feature_;  // equal to cmvn_ if pitch feats are not appended
     delete pitch_feature_;
     delete pitch_;
   }
   delete cmvn_;
   delete base_feature_;
 }

◆ OnlineFeaturePipeline() [2/2]

OnlineFeaturePipeline	(	const OnlineFeaturePipelineConfig &	cfg,
		const Matrix< BaseFloat > &	lda_mat,
		const Matrix< BaseFloat > &	global_cmvn_stats
	)

private

The following constructor is used internally in the New() function; it has the same effect as initializing from just "cfg", but avoids re-reading the LDA transform from disk.

Definition at line 104 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::Init().

                                                :
     config_(config), lda_mat_(lda_mat), global_cmvn_stats_(global_cmvn_stats) {
   Init();
 }

Member Function Documentation

◆ AcceptWaveform()

void AcceptWaveform	(	BaseFloat	sampling_rate,
		const VectorBase< BaseFloat > &	waveform
	)

Accept more data to process (won't actually process it, will just copy it).

sampling_rate is necessary just to assert it equals what's in the config.

Definition at line 272 of file online-feature-pipeline.cc.

References OnlineBaseFeature::AcceptWaveform(), OnlinePitchFeature::AcceptWaveform(), OnlineFeaturePipeline::base_feature_, and OnlineFeaturePipeline::pitch_.

                                            {
   base_feature_->AcceptWaveform(sampling_rate, waveform);
   if (pitch_)
     pitch_->AcceptWaveform(sampling_rate, waveform);
 }

◆ AdaptedFeature()

OnlineFeatureInterface * AdaptedFeature ( ) const

private

returns adapted feature if fmllr_ exists, else UnadaptedFeature().

If this were not private we would have const and non-const versions returning const and non-const pointers.

Definition at line 138 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::fmllr_, and OnlineFeaturePipeline::UnadaptedFeature().

Referenced by OnlineFeaturePipeline::Dim(), OnlineFeaturePipeline::GetFrame(), OnlineFeaturePipeline::IsLastFrame(), and OnlineFeaturePipeline::NumFramesReady().

                                                                     {
   if (fmllr_) return fmllr_;
   else
     return UnadaptedFeature();
 }

◆ Dim()

int32 Dim ( ) const

virtual

Member functions from OnlineFeatureInterface:

Implements OnlineFeatureInterface.

Definition at line 241 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::AdaptedFeature(), and OnlineFeatureInterface::Dim().

Referenced by SingleUtteranceGmmDecoder::EstimateFmllr(), and SingleUtteranceGmmDecoder::GetGaussianPosteriors().

                                        {
   return AdaptedFeature()->Dim();
 }

◆ FrameShiftInSeconds()

BaseFloat FrameShiftInSeconds ( ) const

inlinevirtual

Implements OnlineFeatureInterface.

Definition at line 184 of file online-feature-pipeline.h.

Referenced by SingleUtteranceGmmDecoder::AdvanceDecoding(), and SingleUtteranceGmmDecoder::EndpointDetected().

                                         {
     return config_.FrameShiftInSeconds();
   }

◆ FreezeCmvn()

void FreezeCmvn ( )

Definition at line 237 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::cmvn_, OnlineCmvn::Freeze(), and OnlineCmvn::NumFramesReady().

Referenced by SingleUtteranceGmmDecoder::EstimateFmllr().

                                        {
   cmvn_->Freeze(cmvn_->NumFramesReady() - 1);
 }

◆ GetAsMatrix()

void GetAsMatrix ( Matrix< BaseFloat > * feats )

Definition at line 299 of file online-feature-pipeline.cc.

References OnlineProcessPitch::Dim(), OnlineProcessPitch::GetFrame(), rnnlm::i, OnlineFeaturePipeline::NumFramesReady(), OnlineFeaturePipeline::pitch_, OnlineFeaturePipeline::pitch_feature_, and Matrix< Real >::Resize().

Referenced by SingleUtteranceGmmDecoder::EstimateFmllr().

                                                                 {
   if (pitch_) {
     feats->Resize(NumFramesReady(), pitch_feature_->Dim());
     for (int32 i = 0; i < NumFramesReady(); i++) {
       SubVector<BaseFloat> row(*feats, i);
       pitch_feature_->GetFrame(i, &row);
     }
   }
 }

◆ GetCmvnState()

void GetCmvnState ( OnlineCmvnState * cmvn_state )

Definition at line 149 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::cmvn_, OnlineCmvn::GetState(), and OnlineCmvn::NumFramesReady().

Referenced by SingleUtteranceGmmDecoder::GetAdaptationState().

                                                                     {
   int32 frame = cmvn_->NumFramesReady() - 1;
   // the following call will crash if no frames are ready.
   cmvn_->GetState(frame, cmvn_state);
 }

◆ GetFrame()

void GetFrame	(	int32	frame,
		VectorBase< BaseFloat > *	feat
	)

virtual

Gets the feature vector for this frame.

Before calling this for a given frame, it is assumed that you called NumFramesReady() and it returned a number greater than "frame". Otherwise this call will likely crash with an assert failure. This function is not declared const, in case there is some kind of caching going on, but most of the time it shouldn't modify the class.

Implements OnlineFeatureInterface.

Definition at line 251 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::AdaptedFeature(), and OnlineFeatureInterface::GetFrame().

Referenced by SingleUtteranceGmmDecoder::EstimateFmllr(), and SingleUtteranceGmmDecoder::GetGaussianPosteriors().

                                                                   {
   AdaptedFeature()->GetFrame(frame, feat);
 }

◆ HaveFmllrTransform()

bool HaveFmllrTransform ( )

inline

Definition at line 201 of file online-feature-pipeline.h.

Referenced by SingleUtteranceGmmDecoder::HaveTransform().

201 { return fmllr_ != NULL; }

kaldi::OnlineFeaturePipeline::fmllr_

OnlineFeatureInterface * fmllr_

Definition: online-feature-pipeline.h:245

◆ Init()

void Init ( )

private

Init() is to be called from the constructor; it assumes the pointer members are all uninitialized but config_ and lda_mat_ are initialized.

Definition at line 159 of file online-feature-pipeline.cc.

Referenced by OnlineFeaturePipeline::OnlineFeaturePipeline().

                                  {
   if (config_.feature_type == "mfcc") {
     base_feature_ = new OnlineMfcc(config_.mfcc_opts);
   } else if (config_.feature_type == "plp") {
     base_feature_ = new OnlinePlp(config_.plp_opts);
   } else if (config_.feature_type == "fbank") {
     base_feature_ = new OnlineFbank(config_.fbank_opts);
   } else {
     KALDI_ERR << "Code error: invalid feature type " << config_.feature_type;
   }
 
   {
     KALDI_ASSERT(global_cmvn_stats_.NumRows() != 0);
     if (config_.add_pitch) {
       int32 global_dim = global_cmvn_stats_.NumCols() - 1;
       int32 dim = base_feature_->Dim();
       KALDI_ASSERT(global_dim >= dim);
       if (global_dim > dim) {
         Matrix<BaseFloat> last_col(global_cmvn_stats_.ColRange(global_dim, 1));
         global_cmvn_stats_.Resize(global_cmvn_stats_.NumRows(), dim + 1,
                                   kCopyData);
         global_cmvn_stats_.ColRange(dim, 1).CopyFromMat(last_col);
       }
     }
     Matrix<double> global_cmvn_stats_dbl(global_cmvn_stats_);
     OnlineCmvnState initial_state(global_cmvn_stats_dbl);
     cmvn_ = new OnlineCmvn(config_.cmvn_opts, initial_state, base_feature_);
   }
 
   if (config_.add_pitch) {
     pitch_ = new OnlinePitchFeature(config_.pitch_opts);
     pitch_feature_ = new OnlineProcessPitch(config_.pitch_process_opts,
                                             pitch_);
     feature_ = new OnlineAppendFeature(cmvn_, pitch_feature_);
   } else {
     pitch_ = NULL;
     pitch_feature_ = NULL;
     feature_ = cmvn_;
   }
 
   if (config_.splice_feats && config_.add_deltas) {
     KALDI_ERR << "You cannot supply both --add-deltas and "
               << "--splice-feats options.";
   } else if (config_.splice_feats) {
     splice_or_delta_ = new OnlineSpliceFrames(config_.splice_opts,
                                               feature_);
   } else if (config_.add_deltas) {
     splice_or_delta_ = new OnlineDeltaFeature(config_.delta_opts,
                                               feature_);
   } else {
     splice_or_delta_ = NULL;
   }
 
   if (lda_mat_.NumRows() != 0) {
     lda_ = new OnlineTransform(lda_mat_,
                                (splice_or_delta_ != NULL ?
                                 splice_or_delta_ : feature_));
   } else {
     lda_ = NULL;
   }
 
   fmllr_ = NULL;  // This will be set up if the user calls SetTransform().
 }

◆ InputFinished()

void InputFinished ( )

Definition at line 280 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::base_feature_, OnlineBaseFeature::InputFinished(), OnlinePitchFeature::InputFinished(), and OnlineFeaturePipeline::pitch_.

                                           {
   base_feature_->InputFinished();
   if (pitch_)
     pitch_->InputFinished();
 }

◆ IsLastFrame()

bool IsLastFrame ( int32 frame ) const

virtual

Returns true if this is the last frame.

Frame indices are zero-based, so the first frame is zero. IsLastFrame(-1) will return false, unless the file is empty (which is a case that I'm not sure all the code will handle, so be careful). This function may return false for some frame if we haven't yet decided to terminate decoding, but later true if we decide to terminate decoding. This function exists mainly to correctly handle end effects in feature extraction, and is not a mechanism to determine how many frames are in the decodable object (as it used to be, and for backward compatibility, still is, in the Decodable interface).

Implements OnlineFeatureInterface.

Definition at line 244 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::AdaptedFeature(), and OnlineFeatureInterface::IsLastFrame().

                                                          {
   return AdaptedFeature()->IsLastFrame(frame);
 }

◆ New()

OnlineFeaturePipeline * New ( ) const

returns a newly initialized copy of *this– this does not duplicate all the internal state or the speaker-adaptation state, but gives you a freshly initialized version of this object, as if you had initialized it using the constructor that takes the config file.

After calling this you may want to call SetCmvnState() and SetTransform().

Definition at line 124 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::config_, OnlineFeaturePipeline::global_cmvn_stats_, OnlineFeaturePipeline::lda_mat_, and OnlineFeaturePipeline::OnlineFeaturePipeline().

                                                         {
   return new OnlineFeaturePipeline(config_, lda_mat_,
                                    global_cmvn_stats_);
 }

◆ NumFramesReady()

int32 NumFramesReady ( ) const

virtual

returns the feature dimension.

Returns the total number of frames, since the start of the utterance, that are now available. In an online-decoding context, this will likely increase with time as more data becomes available.

Implements OnlineFeatureInterface.

Definition at line 247 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::AdaptedFeature(), and OnlineFeatureInterface::NumFramesReady().

Referenced by OnlineFeaturePipeline::GetAsMatrix().

                                                   {
   return AdaptedFeature()->NumFramesReady();
 }

◆ SetCmvnState()

void SetCmvnState ( const OnlineCmvnState & cmvn_state )

Set the CMVN state to a particular value (will generally be called after Copy().

Definition at line 145 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::cmvn_, and OnlineCmvn::SetState().

                                                                           {
   cmvn_->SetState(cmvn_state);
 }

◆ SetTransform()

void SetTransform ( const MatrixBase< BaseFloat > & transform )

Definition at line 223 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::fmllr_, MatrixBase< Real >::NumRows(), and OnlineFeaturePipeline::UnadaptedFeature().

Referenced by SingleUtteranceGmmDecoder::EstimateFmllr(), and SingleUtteranceGmmDecoder::SingleUtteranceGmmDecoder().

                                             {
   if (fmllr_ != NULL) {  // we already had a transform;  delete this
     // object.
     delete fmllr_;
     fmllr_ = NULL;
   }
   if (transform.NumRows() != 0) {
     OnlineFeatureInterface *feat = UnadaptedFeature();
     fmllr_ = new OnlineTransform(transform, feat);
   }
 }

◆ UnadaptedFeature()

OnlineFeatureInterface * UnadaptedFeature ( ) const

private

returns lda_ if it exists, else splice_or_delta_, else cmvn_.

If this were not private we would have const and non-const versions returning const and non-const pointers.

Definition at line 129 of file online-feature-pipeline.cc.

References OnlineFeaturePipeline::feature_, KALDI_ASSERT, OnlineFeaturePipeline::lda_, and OnlineFeaturePipeline::splice_or_delta_.

Referenced by OnlineFeaturePipeline::AdaptedFeature(), and OnlineFeaturePipeline::SetTransform().

                                                                       {
   if (lda_) return lda_;
   else if (splice_or_delta_) return splice_or_delta_;
   else {
     KALDI_ASSERT(feature_ != NULL);
     return feature_;
   }
 }

Member Data Documentation

◆ base_feature_

OnlineBaseFeature* base_feature_

private

Definition at line 229 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::AcceptWaveform(), OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::InputFinished(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ cmvn_

OnlineCmvn* cmvn_

private

Definition at line 234 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::FreezeCmvn(), OnlineFeaturePipeline::GetCmvnState(), OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::SetCmvnState(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ config_

OnlineFeaturePipelineConfig config_

private

Definition at line 225 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::Init(), and OnlineFeaturePipeline::New().

◆ feature_

OnlineFeatureInterface* feature_

private

Definition at line 232 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::UnadaptedFeature(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ fmllr_

OnlineFeatureInterface* fmllr_

private

Definition at line 245 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::AdaptedFeature(), OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::SetTransform(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ global_cmvn_stats_

Matrix<BaseFloat> global_cmvn_stats_

private

Definition at line 227 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::New(), and OnlineFeaturePipeline::OnlineFeaturePipeline().

◆ lda_

OnlineFeatureInterface* lda_

private

Definition at line 238 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::UnadaptedFeature(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ lda_mat_

Matrix<BaseFloat> lda_mat_

private

Definition at line 226 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::New(), and OnlineFeaturePipeline::OnlineFeaturePipeline().

◆ pitch_

OnlinePitchFeature* pitch_

private

Definition at line 230 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::AcceptWaveform(), OnlineFeaturePipeline::GetAsMatrix(), OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::InputFinished(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ pitch_feature_

OnlineProcessPitch* pitch_feature_

private

Definition at line 231 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::GetAsMatrix(), OnlineFeaturePipeline::Init(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

◆ splice_or_delta_

OnlineFeatureInterface* splice_or_delta_

private

Definition at line 235 of file online-feature-pipeline.h.

Referenced by OnlineFeaturePipeline::Init(), OnlineFeaturePipeline::UnadaptedFeature(), and OnlineFeaturePipeline::~OnlineFeaturePipeline().

The documentation for this class was generated from the following files:

online2/online-feature-pipeline.h
online2/online-feature-pipeline.cc

Private Attributes
OnlineFeaturePipelineConfig	config_

Matrix< BaseFloat >	lda_mat_

Matrix< BaseFloat >	global_cmvn_stats_

OnlineBaseFeature *	base_feature_

OnlinePitchFeature *	pitch_

OnlineProcessPitch *	pitch_feature_

OnlineFeatureInterface *	feature_

OnlineCmvn *	cmvn_

OnlineFeatureInterface *	splice_or_delta_

OnlineFeatureInterface *	lda_

OnlineFeatureInterface *	fmllr_

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ OnlineFeaturePipeline() [1/2]

◆ ~OnlineFeaturePipeline()

◆ OnlineFeaturePipeline() [2/2]

Member Function Documentation

◆ AcceptWaveform()

◆ AdaptedFeature()

◆ Dim()

◆ FrameShiftInSeconds()

◆ FreezeCmvn()

◆ GetAsMatrix()

◆ GetCmvnState()

◆ GetFrame()

◆ HaveFmllrTransform()

◆ Init()

◆ InputFinished()

◆ IsLastFrame()

◆ New()

◆ NumFramesReady()

◆ SetCmvnState()

◆ SetTransform()

◆ UnadaptedFeature()

Member Data Documentation

◆ base_feature_

◆ cmvn_

◆ config_

◆ feature_

◆ fmllr_

◆ global_cmvn_stats_

◆ lda_

◆ lda_mat_

◆ pitch_

◆ pitch_feature_

◆ splice_or_delta_