kaldi::nnet3::utterance_splitting Namespace Reference

This namespace contains things needed for the implementation of the function NnetBatchComputer::SplitUtteranceIntoTasks(). More...

Functions
void	GetOutputFrameInfoForTasks (const NnetBatchComputerOptions &opts, int32 num_subsampled_frames, int32 num_subsampled_frames_per_chunk, std::vector< NnetInferenceTask > *tasks)
	This function figures out how many chunks are needed for this utterance, sets 'tasks' to a vector with that many elements, and sets up the following elements in 'tasks': output_t_stride num_output_frames num_initial_unused_output_frames num_used_output_frames. More...
void	AddOnlineIvectorsToTasks (const NnetBatchComputerOptions &opts, const CuMatrix< BaseFloat > &online_ivectors, int32 online_ivector_period, std::vector< NnetInferenceTask > *tasks)
static void	SplitInputToTasks (const NnetBatchComputerOptions &opts, int32 nnet_left_context, int32 nnet_right_context, const CuMatrix< BaseFloat > &input, std::vector< NnetInferenceTask > *tasks)
	This function sets up the 'input' and 'first_input_t' and 'is_edge' members of the 'tasks' array; it is responsible for working out, for each task, which input frames it needs (including left-context and right-context). More...

Detailed Description

This namespace contains things needed for the implementation of the function NnetBatchComputer::SplitUtteranceIntoTasks().

Function Documentation

AddOnlineIvectorsToTasks()

void kaldi::nnet3::utterance_splitting::AddOnlineIvectorsToTasks	(	const NnetBatchComputerOptions &	opts,
		const CuMatrix< BaseFloat > &	online_ivectors,
		int32	online_ivector_period,
		std::vector< NnetInferenceTask > *	tasks
	)

Definition at line 729 of file nnet-batch-compute.cc.

References NnetInferenceTask::first_used_output_frame_index, NnetSimpleComputationOptions::frame_subsampling_factor, rnnlm::i, NnetInferenceTask::ivector, KALDI_ERR, NnetInferenceTask::num_initial_unused_output_frames, NnetInferenceTask::num_output_frames, CuMatrixBase< Real >::NumRows(), and CuMatrixBase< Real >::Row().

Referenced by NnetBatchComputer::SplitUtteranceIntoTasks().

                                         {
  int32 f = opts.frame_subsampling_factor,
      num_tasks = tasks->size();
  for (int32 i = 0; i < num_tasks; i++) {
    NnetInferenceTask &task = (*tasks)[i];
    // begin_output_t and end_output_t are the subsampled frame indexes at
    // the output; you'd have to multiply them by f to get real frame indexes.
    int32 begin_output_t = task.first_used_output_frame_index -
        task.num_initial_unused_output_frames,
        mid_output_t = begin_output_t + (task.num_output_frames / 2),
        mid_input_t = mid_output_t * f,
        ivector_frame = mid_input_t / online_ivector_period,
        num_ivector_frames = online_ivectors.NumRows(),
        margin_in_frames = 20,
        margin_in_ivector_frames =
        (margin_in_frames + online_ivector_period - 1) / online_ivector_period;
    // the 'margin' is our tolerance for when the number of rows of
    // 'online_ivectors' is less than what we expected; we allow 20 frames of
    // tolerance in the numbering of the original (input) features.
    if (ivector_frame >= num_ivector_frames) {
      if (num_ivector_frames > 0 && ivector_frame > num_ivector_frames -
          margin_in_ivector_frames) {
        ivector_frame = num_ivector_frames - 1;  // Just take the last available one.
      } else {
        KALDI_ERR << "Could not get iVector for frame " << ivector_frame
                  << ", online-ivectors matrix has "
                  << online_ivectors.NumRows()
                  << " rows.  Mismatched --online-ivector-period?";
      }
    }
    task.ivector = online_ivectors.Row(ivector_frame);
  }
}

GetOutputFrameInfoForTasks()

void kaldi::nnet3::utterance_splitting::GetOutputFrameInfoForTasks	(	const NnetBatchComputerOptions &	opts,
		int32	num_subsampled_frames,
		int32	num_subsampled_frames_per_chunk,
		std::vector< NnetInferenceTask > *	tasks
	)

This function figures out how many chunks are needed for this utterance, sets 'tasks' to a vector with that many elements, and sets up the following elements in 'tasks': output_t_stride num_output_frames num_initial_unused_output_frames num_used_output_frames.

Parameters

[in]	opts	Options class
[in]	num_subsampled_frames	The number of output frames in this utterance. Must be > 0.
[in]	num_subsampled_frames_per_chunk	The number of output frames per chunk
[out]	The	'tasks' array is output to here; it will have one task per chunk, with only the members 'output_t_stride', 'num_output_frames', 'num_initial_unused_output_frames', 'num_used_output_frames' and 'is_irregular' set up.

Definition at line 661 of file nnet-batch-compute.cc.

References NnetBatchComputerOptions::ensure_exact_final_context, NnetInferenceTask::first_used_output_frame_index, NnetSimpleComputationOptions::frame_subsampling_factor, rnnlm::i, NnetInferenceTask::is_irregular, KALDI_ASSERT, NnetInferenceTask::num_initial_unused_output_frames, NnetInferenceTask::num_output_frames, and NnetInferenceTask::num_used_output_frames.

Referenced by NnetBatchComputer::SplitUtteranceIntoTasks().

                                         {
  KALDI_ASSERT(num_subsampled_frames > 0);
  int32 fpc = num_subsampled_frames_per_chunk;
  int32 num_tasks = (num_subsampled_frames + fpc - 1) / fpc;
  tasks->resize(num_tasks);
  for (int32 i = 0; i < num_tasks; i++) {
    (*tasks)[i].output_t_stride = opts.frame_subsampling_factor;
  }
  if (num_subsampled_frames <= fpc) {  // there is one chunk.
    KALDI_ASSERT(num_tasks == 1);  // TODO: remove this.
    NnetInferenceTask &task = (*tasks)[0];
    task.first_used_output_frame_index = 0;
    if (opts.ensure_exact_final_context) {
      task.num_output_frames = num_subsampled_frames;
      task.num_initial_unused_output_frames = 0;
      task.num_used_output_frames = num_subsampled_frames;
      task.is_irregular = true;
    } else {
      task.num_output_frames = fpc;
      task.num_initial_unused_output_frames = 0;
      task.num_used_output_frames = num_subsampled_frames;
      task.is_irregular = false;
    }
  } else {
    for (int32 i = 0; i + 1 < num_tasks; i++) {
      NnetInferenceTask &task = (*tasks)[i];
      task.num_output_frames = fpc;
      task.num_initial_unused_output_frames = 0;
      task.num_used_output_frames = fpc;
      task.first_used_output_frame_index = i * fpc;
      task.is_irregular = false;
    }
    // The last chunk will end on the last frame of the file, but we won't use
    // the part of its output that overlaps with the preceding chunk.
    NnetInferenceTask &task = (*tasks)[num_tasks - 1];
    task.num_output_frames = fpc;
    task.num_initial_unused_output_frames = ((num_tasks - 1) * fpc) -
        (num_subsampled_frames - fpc);
    task.num_used_output_frames =
        num_subsampled_frames - ((num_tasks - 1) * fpc);
    task.first_used_output_frame_index = (num_tasks - 1) * fpc;
    task.is_irregular = false;
  }

  if (true) {
    // Do some checking.  TODO: remove this.
    KALDI_ASSERT((*tasks)[0].first_used_output_frame_index == 0);
    for (int32 i = 1; i < num_tasks; i++) {
      KALDI_ASSERT((*tasks)[i].first_used_output_frame_index ==
                   (*tasks)[i-1].first_used_output_frame_index +
                   (*tasks)[i-1].num_used_output_frames);
    }
    KALDI_ASSERT((*tasks)[num_tasks-1].first_used_output_frame_index +
                 (*tasks)[num_tasks-1].num_used_output_frames ==
                 num_subsampled_frames);
    for (int32 i = 0; i < num_tasks; i++) {
      const NnetInferenceTask &task = (*tasks)[i];
      KALDI_ASSERT(task.num_used_output_frames +
                   task.num_initial_unused_output_frames <=
                   task.num_output_frames);
    }
  }
}

SplitInputToTasks()

static void kaldi::nnet3::utterance_splitting::SplitInputToTasks ( const NnetBatchComputerOptions &  opts, int32  nnet_left_context, int32  nnet_right_context, const CuMatrix< BaseFloat > &  input, std::vector< NnetInferenceTask > *  tasks  )

static

This function sets up the 'input' and 'first_input_t' and 'is_edge' members of the 'tasks' array; it is responsible for working out, for each task, which input frames it needs (including left-context and right-context).

The 'nnet_left_context' and 'nnet_right_context' are the inherent left and right context of the network (num-frames required on left and right to compute an output frame), and may be computed by doing: ComputeSimpleNnetContext(nnet, &nnet_left_context_, &nnet_right_context_)

Definition at line 779 of file nnet-batch-compute.cc.

References NnetSimpleComputationOptions::extra_left_context, NnetSimpleComputationOptions::extra_left_context_initial, NnetSimpleComputationOptions::extra_right_context, NnetSimpleComputationOptions::extra_right_context_final, NnetInferenceTask::first_input_t, NnetInferenceTask::first_used_output_frame_index, NnetSimpleComputationOptions::frame_subsampling_factor, rnnlm::i, NnetInferenceTask::input, NnetInferenceTask::is_edge, kaldi::kUndefined, NnetInferenceTask::num_initial_unused_output_frames, NnetInferenceTask::num_output_frames, CuMatrixBase< Real >::NumCols(), and CuMatrixBase< Real >::NumRows().

Referenced by NnetBatchComputer::SplitUtteranceIntoTasks().

                                                                   {
  int32 num_input_frames = input.NumRows(),
      f = opts.frame_subsampling_factor,
      num_subsampled_frames = (num_input_frames + f - 1) / f,
      extra_left_context_initial = (opts.extra_left_context_initial < 0 ?
                                    opts.extra_left_context :
                                    opts.extra_left_context_initial),
      extra_right_context_final = (opts.extra_right_context_final < 0 ?
                                   opts.extra_right_context :
                                   opts.extra_right_context_final),
      num_tasks = tasks->size();

  for (int32 i = 0; i < num_tasks; i++) {
    NnetInferenceTask &task = (*tasks)[i];
    // begin_output_t and end_output_t are the subsampled frame indexes at
    // the output; you'd have to multiply them by f to get real frame indexes.
    int32 begin_output_t = task.first_used_output_frame_index -
        task.num_initial_unused_output_frames,
        end_output_t = begin_output_t + task.num_output_frames;
    // begin_input_t and end_input_t are the real 't' values corresponding to
    // begin_output_t and end_output_t; they are the beginning and end
    // (i.e. first and last-plus-one) frame indexes without any left or right
    // context.
    int32 begin_input_t = begin_output_t * f,
        end_input_t = end_output_t * f;
    // Detect whether the left and right edges touch (or pass over) the left
    // and right boundaries.  Note: we don't expect begin_output_t to ever be
    // negative.
    bool left_edge = (begin_output_t <= 0),
        right_edge = (end_output_t >= num_subsampled_frames);
    int32 tot_left_context = nnet_left_context +
        (left_edge ? extra_left_context_initial : opts.extra_left_context),
        tot_right_context = nnet_right_context +
        (right_edge ? extra_right_context_final : opts.extra_right_context);

    // 'is_edge' is only true if it's an edge minibatch *and* its being an
    // edge actually made a difference to the structure of the example.
    task.is_edge =
        (tot_left_context != nnet_left_context + opts.extra_left_context ||
         tot_right_context !=  nnet_right_context + opts.extra_right_context);

    int32 begin_input_t_padded = begin_input_t - tot_left_context,
        end_input_t_padded = end_input_t + tot_right_context;

    // 'task.first_input_t' is a representation of 'begin_input_t_padded' in a
    // shifted/normalized numbering where the output time indexes start from
    // zero.
    task.first_input_t = begin_input_t_padded - (begin_output_t * f);

    task.input.Resize(end_input_t_padded - begin_input_t_padded,
                      input.NumCols(), kUndefined);

    // Copy from intput into task input with clamping
    task.input.CopyRangeFromMatClamped(input, begin_input_t_padded, 
        end_input_t_padded, 0, num_input_frames-1);
  }
}