This online-feature class implements post processing of pitch features. More...

#include <pitch-functions.h>

Inheritance diagram for OnlineProcessPitch:

Collaboration diagram for OnlineProcessPitch:

[legend]

Classes
struct	NormalizationStats

Public Member Functions
virtual int32	Dim () const

virtual bool	IsLastFrame (int32 frame) const
	Returns true if this is the last frame. More...

virtual BaseFloat	FrameShiftInSeconds () const

virtual int32	NumFramesReady () const
	returns the feature dimension. More...

virtual void	GetFrame (int32 frame, VectorBase< BaseFloat > *feat)
	Gets the feature vector for this frame. More...

virtual	~OnlineProcessPitch ()

	OnlineProcessPitch (const ProcessPitchOptions &opts, OnlineFeatureInterface *src)
	Note on the implementation of OnlineProcessPitch: the OnlineFeatureInterface allows random access to features (i.e. More...

Public Member Functions inherited from OnlineFeatureInterface
virtual void	GetFrames (const std::vector< int32 > &frames, MatrixBase< BaseFloat > *feats)
	This is like GetFrame() but for a collection of frames. More...

virtual	~OnlineFeatureInterface ()
	Virtual destructor. More...

Private Types
enum	{ kRawFeatureDim = 2 }

Private Member Functions
BaseFloat	GetPovFeature (int32 frame) const
	Computes and returns the POV feature for this frame. More...

BaseFloat	GetDeltaPitchFeature (int32 frame)
	Computes and returns the delta-log-pitch feature for this frame. More...

BaseFloat	GetRawLogPitchFeature (int32 frame) const
	Computes and returns the raw log-pitch feature for this frame. More...

BaseFloat	GetNormalizedLogPitchFeature (int32 frame)
	Computes and returns the mean-subtracted log-pitch feature for this frame. More...

void	GetNormalizationWindow (int32 frame, int32 src_frames_ready, int32 window_begin, int32 window_end) const
	Computes the normalization window sizes. More...

void	UpdateNormalizationStats (int32 frame)
	Makes sure the entry in normalization_stats_ for this frame is up to date; called from GetNormalizedLogPitchFeature. More...

Private Attributes
ProcessPitchOptions	opts_

OnlineFeatureInterface *	src_

int32	dim_

std::vector< BaseFloat >	delta_feature_noise_

std::vector< NormalizationStats >	normalization_stats_

Detailed Description

This online-feature class implements post processing of pitch features.

Inputs are original 2 dims (nccf, pitch). It can produce various kinds of outputs, using the default options it will be (pov-feature, normalized-log-pitch, delta-log-pitch).

Definition at line 332 of file pitch-functions.h.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum

private

Enumerator
kRawFeatureDim

Definition at line 359 of file pitch-functions.h.

359 { kRawFeatureDim = 2}; // anonymous enum to define a constant.

kaldi::OnlineProcessPitch::kRawFeatureDim

Definition: pitch-functions.h:359

Constructor & Destructor Documentation

◆ ~OnlineProcessPitch()

virtual ~OnlineProcessPitch ( )

inlinevirtual

Definition at line 352 of file pitch-functions.h.

352 { }

◆ OnlineProcessPitch()

OnlineProcessPitch	(	const ProcessPitchOptions &	opts,
		OnlineFeatureInterface *	src
	)

Note on the implementation of OnlineProcessPitch: the OnlineFeatureInterface allows random access to features (i.e.

not necessarily sequential order), so we need to support that. But we don't need to support it very efficiently, and our implementation is most efficient if frames are accessed in sequential order.

Also note: we have to be a bit careful in this implementation because the input features may change. That is: if we call src_->GetFrame(t, &vec) from GetFrame(), we can't guarantee that a later call to src_->GetFrame(t, &vec) from another GetFrame() will return the same value. In fact, while designing this class we used some knowledge of how the OnlinePitchFeature class works to minimize the amount of re-querying we had to do.

Definition at line 1398 of file pitch-functions.cc.

References OnlineFeatureInterface::Dim(), OnlineProcessPitch::dim_, KALDI_ASSERT, and OnlineProcessPitch::kRawFeatureDim.

                                 :
     opts_(opts), src_(src),
     dim_ ((opts.add_pov_feature ? 1 : 0)
           + (opts.add_normalized_log_pitch ? 1 : 0)
           + (opts.add_delta_pitch ? 1 : 0)
           + (opts.add_raw_log_pitch ? 1 : 0)) {
   KALDI_ASSERT(dim_ > 0 &&
                " At least one of the pitch features should be chosen. "
                "Check your post-process-pitch options.");
   KALDI_ASSERT(src->Dim() == kRawFeatureDim &&
                "Input feature must be pitch feature (should have dimension 2)");
 }

Member Function Documentation

◆ Dim()

virtual int32 Dim ( ) const

inlinevirtual

Implements OnlineFeatureInterface.

Definition at line 334 of file pitch-functions.h.

Referenced by kaldi::ComputeAndProcessKaldiPitch(), OnlineFeaturePipeline::GetAsMatrix(), kaldi::ProcessPitch(), kaldi::UnitTestDelay(), and kaldi::UnitTestPieces().

334 { return dim_; }

kaldi::OnlineProcessPitch::dim_

int32 dim_

Definition: pitch-functions.h:365

◆ FrameShiftInSeconds()

virtual BaseFloat FrameShiftInSeconds ( ) const

inlinevirtual

Implements OnlineFeatureInterface.

Definition at line 344 of file pitch-functions.h.

                                                 {
     return src_->FrameShiftInSeconds();
   }

◆ GetDeltaPitchFeature()

BaseFloat GetDeltaPitchFeature ( int32 frame )

inlineprivate

Computes and returns the delta-log-pitch feature for this frame.

Called from GetFrame().

Definition at line 1439 of file pitch-functions.cc.

References kaldi::ComputeDeltas(), OnlineProcessPitch::delta_feature_noise_, ProcessPitchOptions::delta_pitch_noise_stddev, ProcessPitchOptions::delta_pitch_scale, ProcessPitchOptions::delta_window, OnlineProcessPitch::GetRawLogPitchFeature(), OnlineFeatureInterface::NumFramesReady(), OnlineProcessPitch::opts_, DeltaFeaturesOptions::order, kaldi::RandGauss(), OnlineProcessPitch::src_, and DeltaFeaturesOptions::window.

Referenced by OnlineProcessPitch::GetFrame().

                                                               {
   // Rather than computing the delta pitch directly in code here,
   // which might seem easier, we accumulate a small window of features
   // and call ComputeDeltas.  This might seem like overkill; the reason
   // we do it this way is to ensure that the end effects (at file
   // beginning and end) are handled in a consistent way.
   int32 context = opts_.delta_window;
   int32 start_frame = std::max(0, frame - context),
       end_frame = std::min(frame + context + 1, src_->NumFramesReady()),
       frames_in_window = end_frame - start_frame;
   Matrix<BaseFloat> feats(frames_in_window, 1),
       delta_feats;
 
   for (int32 f = start_frame; f < end_frame; f++)
     feats(f - start_frame, 0) = GetRawLogPitchFeature(f);
 
   DeltaFeaturesOptions delta_opts;
   delta_opts.order = 1;
   delta_opts.window = opts_.delta_window;
   ComputeDeltas(delta_opts, feats, &delta_feats);
   while (delta_feature_noise_.size() <= static_cast<size_t>(frame)) {
     delta_feature_noise_.push_back(RandGauss() *
                                    opts_.delta_pitch_noise_stddev);
   }
   // note: delta_feats will have two columns, second contains deltas.
   return (delta_feats(frame - start_frame, 1) + delta_feature_noise_[frame]) *
       opts_.delta_pitch_scale;
 }

◆ GetFrame()

void GetFrame	(	int32	frame,
		VectorBase< BaseFloat > *	feat
	)

virtual

Gets the feature vector for this frame.

Before calling this for a given frame, it is assumed that you called NumFramesReady() and it returned a number greater than "frame". Otherwise this call will likely crash with an assert failure. This function is not declared const, in case there is some kind of caching going on, but most of the time it shouldn't modify the class.

Implements OnlineFeatureInterface.

Definition at line 1414 of file pitch-functions.cc.

References ProcessPitchOptions::add_delta_pitch, ProcessPitchOptions::add_normalized_log_pitch, ProcessPitchOptions::add_pov_feature, ProcessPitchOptions::add_raw_log_pitch, ProcessPitchOptions::delay, VectorBase< Real >::Dim(), OnlineProcessPitch::dim_, OnlineProcessPitch::GetDeltaPitchFeature(), OnlineProcessPitch::GetNormalizedLogPitchFeature(), OnlineProcessPitch::GetPovFeature(), OnlineProcessPitch::GetRawLogPitchFeature(), KALDI_ASSERT, OnlineProcessPitch::NumFramesReady(), and OnlineProcessPitch::opts_.

Referenced by kaldi::ComputeAndProcessKaldiPitch(), OnlineFeaturePipeline::GetAsMatrix(), kaldi::ProcessPitch(), kaldi::UnitTestDelay(), and kaldi::UnitTestPieces().

                                                                {
   int32 frame_delayed = frame < opts_.delay ? 0 : frame - opts_.delay;
   KALDI_ASSERT(feat->Dim() == dim_ &&
                frame_delayed < NumFramesReady());
   int32 index = 0;
   if (opts_.add_pov_feature)
     (*feat)(index++) = GetPovFeature(frame_delayed);
   if (opts_.add_normalized_log_pitch)
     (*feat)(index++) = GetNormalizedLogPitchFeature(frame_delayed);
   if (opts_.add_delta_pitch)
     (*feat)(index++) = GetDeltaPitchFeature(frame_delayed);
   if (opts_.add_raw_log_pitch)
     (*feat)(index++) = GetRawLogPitchFeature(frame_delayed);
   KALDI_ASSERT(index == dim_);
 }

◆ GetNormalizationWindow()

void GetNormalizationWindow	(	int32	frame,
		int32	src_frames_ready,
		int32 *	window_begin,
		int32 *	window_end
	)		const

inlineprivate

Computes the normalization window sizes.

Definition at line 1487 of file pitch-functions.cc.

References ProcessPitchOptions::normalization_left_context, ProcessPitchOptions::normalization_right_context, and OnlineProcessPitch::opts_.

Referenced by OnlineProcessPitch::UpdateNormalizationStats().

                                                                          {
   int32 left_context = opts_.normalization_left_context;
   int32 right_context = opts_.normalization_right_context;
   *window_begin = std::max(0, t - left_context);
   *window_end = std::min(t + right_context + 1, src_frames_ready);
 }

◆ GetNormalizedLogPitchFeature()

BaseFloat GetNormalizedLogPitchFeature ( int32 frame )

inlineprivate

Computes and returns the mean-subtracted log-pitch feature for this frame.

Called from GetFrame().

Definition at line 1476 of file pitch-functions.cc.

References OnlineProcessPitch::GetRawLogPitchFeature(), OnlineProcessPitch::normalization_stats_, OnlineProcessPitch::opts_, ProcessPitchOptions::pitch_scale, and OnlineProcessPitch::UpdateNormalizationStats().

Referenced by OnlineProcessPitch::GetFrame().

                                                                       {
   UpdateNormalizationStats(frame);
   BaseFloat log_pitch = GetRawLogPitchFeature(frame),
       avg_log_pitch = normalization_stats_[frame].sum_log_pitch_pov /
         normalization_stats_[frame].sum_pov,
       normalized_log_pitch = log_pitch - avg_log_pitch;
   return normalized_log_pitch * opts_.pitch_scale;
 }

◆ GetPovFeature()

BaseFloat GetPovFeature ( int32 frame ) const

inlineprivate

Computes and returns the POV feature for this frame.

Called from GetFrame().

Definition at line 1431 of file pitch-functions.cc.

References OnlineFeatureInterface::GetFrame(), OnlineProcessPitch::kRawFeatureDim, kaldi::NccfToPovFeature(), OnlineProcessPitch::opts_, ProcessPitchOptions::pov_offset, ProcessPitchOptions::pov_scale, and OnlineProcessPitch::src_.

Referenced by OnlineProcessPitch::GetFrame().

                                                              {
   Vector<BaseFloat> tmp(kRawFeatureDim);
   src_->GetFrame(frame, &tmp);  // (NCCF, pitch) from pitch extractor
   BaseFloat nccf = tmp(0);
   return opts_.pov_scale * NccfToPovFeature(nccf)
       + opts_.pov_offset;
 }

◆ GetRawLogPitchFeature()

BaseFloat GetRawLogPitchFeature ( int32 frame ) const

inlineprivate

Computes and returns the raw log-pitch feature for this frame.

Called from GetFrame().

Definition at line 1468 of file pitch-functions.cc.

References OnlineFeatureInterface::GetFrame(), KALDI_ASSERT, OnlineProcessPitch::kRawFeatureDim, kaldi::Log(), and OnlineProcessPitch::src_.

Referenced by OnlineProcessPitch::GetDeltaPitchFeature(), OnlineProcessPitch::GetFrame(), and OnlineProcessPitch::GetNormalizedLogPitchFeature().

                                                                      {
   Vector<BaseFloat> tmp(kRawFeatureDim);
   src_->GetFrame(frame, &tmp);
   BaseFloat pitch = tmp(1);
   KALDI_ASSERT(pitch > 0);
   return Log(pitch);
 }

◆ IsLastFrame()

virtual bool IsLastFrame ( int32 frame ) const

inlinevirtual

Returns true if this is the last frame.

Frame indices are zero-based, so the first frame is zero. IsLastFrame(-1) will return false, unless the file is empty (which is a case that I'm not sure all the code will handle, so be careful). This function may return false for some frame if we haven't yet decided to terminate decoding, but later true if we decide to terminate decoding. This function exists mainly to correctly handle end effects in feature extraction, and is not a mechanism to determine how many frames are in the decodable object (as it used to be, and for backward compatibility, still is, in the Decodable interface).

Implements OnlineFeatureInterface.

Definition at line 336 of file pitch-functions.h.

                                               {
     if (frame <= -1)
       return src_->IsLastFrame(-1);
     else if (frame < opts_.delay)
       return src_->IsLastFrame(-1) == true ? false : src_->IsLastFrame(0);
     else
       return src_->IsLastFrame(frame - opts_.delay);
   }

◆ NumFramesReady()

int32 NumFramesReady ( ) const

virtual

returns the feature dimension.

Returns the total number of frames, since the start of the utterance, that are now available. In an online-decoding context, this will likely increase with time as more data becomes available.

Implements OnlineFeatureInterface.

Definition at line 1569 of file pitch-functions.cc.

References ProcessPitchOptions::delay, OnlineFeatureInterface::IsLastFrame(), ProcessPitchOptions::normalization_right_context, OnlineFeatureInterface::NumFramesReady(), OnlineProcessPitch::opts_, and OnlineProcessPitch::src_.

Referenced by kaldi::ComputeAndProcessKaldiPitch(), OnlineProcessPitch::GetFrame(), kaldi::ProcessPitch(), and kaldi::UnitTestDelay().

                                                {
   int32 src_frames_ready = src_->NumFramesReady();
   if (src_frames_ready == 0) {
     return 0;
   } else if (src_->IsLastFrame(src_frames_ready - 1)) {
     return src_frames_ready + opts_.delay;
   } else {
     return std::max(0, src_frames_ready -
       opts_.normalization_right_context + opts_.delay);
   }
 }

◆ UpdateNormalizationStats()

void UpdateNormalizationStats ( int32 frame )

inlineprivate

Makes sure the entry in normalization_stats_ for this frame is up to date; called from GetNormalizedLogPitchFeature.

Definition at line 1502 of file pitch-functions.cc.

Referenced by OnlineProcessPitch::GetNormalizedLogPitchFeature().

                                                              {
   KALDI_ASSERT(frame >= 0);
   if (normalization_stats_.size() <= frame)
     normalization_stats_.resize(frame + 1);
   int32 cur_num_frames = src_->NumFramesReady();
   bool input_finished = src_->IsLastFrame(cur_num_frames - 1);
 
   NormalizationStats &this_stats = normalization_stats_[frame];
   if (this_stats.cur_num_frames == cur_num_frames &&
       this_stats.input_finished == input_finished) {
     // Stats are fully up-to-date.
     return;
   }
   int32 this_window_begin, this_window_end;
   GetNormalizationWindow(frame, cur_num_frames,
                          &this_window_begin, &this_window_end);
 
   if (frame > 0) {
     const NormalizationStats &prev_stats = normalization_stats_[frame - 1];
     if (prev_stats.cur_num_frames == cur_num_frames &&
         prev_stats.input_finished == input_finished) {
       // we'll derive this_stats efficiently from prev_stats.
       // Checking that cur_num_frames and input_finished have not changed
       // ensures that the underlying features will not have changed.
       this_stats = prev_stats;
       int32 prev_window_begin, prev_window_end;
       GetNormalizationWindow(frame - 1, cur_num_frames,
                              &prev_window_begin, &prev_window_end);
       if (this_window_begin != prev_window_begin) {
         KALDI_ASSERT(this_window_begin == prev_window_begin + 1);
         Vector<BaseFloat> tmp(kRawFeatureDim);
         src_->GetFrame(prev_window_begin, &tmp);
         BaseFloat accurate_pov = NccfToPov(tmp(0)),
             log_pitch = Log(tmp(1));
         this_stats.sum_pov -= accurate_pov;
         this_stats.sum_log_pitch_pov -= accurate_pov * log_pitch;
       }
       if (this_window_end != prev_window_end) {
         KALDI_ASSERT(this_window_end == prev_window_end + 1);
         Vector<BaseFloat> tmp(kRawFeatureDim);
         src_->GetFrame(prev_window_end, &tmp);
         BaseFloat accurate_pov = NccfToPov(tmp(0)),
             log_pitch = Log(tmp(1));
         this_stats.sum_pov += accurate_pov;
         this_stats.sum_log_pitch_pov += accurate_pov * log_pitch;
       }
       return;
     }
   }
   // The way we do it here is not the most efficient way to do it;
   // we'll see if it becomes a problem.  The issue is we have to redo
   // this computation from scratch each time we process a new chunk, which
   // may be a little inefficient if the chunk-size is very small.
   this_stats.cur_num_frames = cur_num_frames;
   this_stats.input_finished = input_finished;
   this_stats.sum_pov = 0.0;
   this_stats.sum_log_pitch_pov = 0.0;
   Vector<BaseFloat> tmp(kRawFeatureDim);
   for (int32 f = this_window_begin; f < this_window_end; f++) {
     src_->GetFrame(f, &tmp);
     BaseFloat accurate_pov = NccfToPov(tmp(0)),
         log_pitch = Log(tmp(1));
     this_stats.sum_pov += accurate_pov;
     this_stats.sum_log_pitch_pov += accurate_pov * log_pitch;
   }
 }