This class implements a simplified interface to class NnetBatchComputer, which is suitable for programs like 'nnet3-compute' where you want to support fast GPU-based inference on a sequence of utterances, and get them back from the object in the same order. More...

#include <nnet-batch-compute.h>

Collaboration diagram for NnetBatchInference:

[legend]

Classes
struct	UtteranceInfo

Public Member Functions
	NnetBatchInference (const NnetBatchComputerOptions &opts, const Nnet &nnet, const VectorBase< BaseFloat > &priors)

void	AcceptInput (const std::string &utterance_id, const Matrix< BaseFloat > &input, const Vector< BaseFloat > ivector, const Matrix< BaseFloat > online_ivectors, int32 online_ivector_period)
	The user should call this one by one for the utterances that this class needs to compute (interspersed with calls to GetOutput()). More...

void	Finished ()
	The user should call this after the last input has been provided via AcceptInput(). More...

bool	GetOutput (std::string utterance_id, Matrix< BaseFloat > output)
	The user should call this to obtain output. More...

	~NnetBatchInference ()

Private Member Functions
	KALDI_DISALLOW_COPY_AND_ASSIGN (NnetBatchInference)

void	Compute ()

Static Private Member Functions
static void	ComputeFunc (NnetBatchInference *object)

Private Attributes
NnetBatchComputer	computer_

bool	is_finished_

Semaphore	tasks_ready_semaphore_

std::list< UtteranceInfo * >	utts_

int32	utterance_counter_

std::thread	compute_thread_

Detailed Description

This class implements a simplified interface to class NnetBatchComputer, which is suitable for programs like 'nnet3-compute' where you want to support fast GPU-based inference on a sequence of utterances, and get them back from the object in the same order.

Definition at line 502 of file nnet-batch-compute.h.

Constructor & Destructor Documentation

◆ NnetBatchInference()

NnetBatchInference	(	const NnetBatchComputerOptions &	opts,
		const Nnet &	nnet,
		const VectorBase< BaseFloat > &	priors
	)

Definition at line 1083 of file nnet-batch-compute.cc.

References NnetBatchInference::compute_thread_, and NnetBatchInference::ComputeFunc().

                                         :
     computer_(opts, nnet, priors),
     is_finished_(false),
     utterance_counter_(0) {
   // 'thread_' will run the Compute() function in the background.
   compute_thread_ = std::thread(ComputeFunc, this);
 }

◆ ~NnetBatchInference()

~NnetBatchInference ( )

Definition at line 1157 of file nnet-batch-compute.cc.

References NnetBatchInference::compute_thread_, NnetBatchInference::is_finished_, KALDI_ERR, and NnetBatchInference::utts_.

                                         {
   if (!is_finished_)
     KALDI_ERR << "Object destroyed before Finished() was called.";
   if (!utts_.empty())
     KALDI_ERR << "You should get all output before destroying this object.";
   compute_thread_.join();
 }

Member Function Documentation

◆ AcceptInput()

void AcceptInput	(	const std::string &	utterance_id,
		const Matrix< BaseFloat > &	input,
		const Vector< BaseFloat > *	ivector,
		const Matrix< BaseFloat > *	online_ivectors,
		int32	online_ivector_period
	)

The user should call this one by one for the utterances that this class needs to compute (interspersed with calls to GetOutput()).

This call will block when enough ready-to-be-computed data is present.

Parameters

[in]	utterance_id	The string representing the utterance-id; it will be provided back to the user when GetOutput() is called.
[in]	input	The input features (e.g. MFCCs)
[in]	ivector	If non-NULL, this is expected to be the i-vector for this utterance (and 'online_ivectors' should be NULL).
[in]	online_ivector_period	Only relevant if 'online_ivector' is non-NULL, this says how many frames of 'input' is covered by each row of 'online_ivectors'.

Definition at line 1095 of file nnet-batch-compute.cc.

References NnetBatchComputer::AcceptTask(), NnetBatchInference::computer_, rnnlm::i, NnetBatchInference::UtteranceInfo::num_tasks_finished, Semaphore::Signal(), NnetBatchComputer::SplitUtteranceIntoTasks(), NnetBatchInference::UtteranceInfo::tasks, NnetBatchInference::tasks_ready_semaphore_, NnetBatchInference::utterance_counter_, NnetBatchInference::UtteranceInfo::utterance_id, and NnetBatchInference::utts_.

Referenced by main().

                                  {
 
   UtteranceInfo *info = new UtteranceInfo();
   info->utterance_id = utterance_id;
   info->num_tasks_finished = 0;
   bool output_to_cpu = true;  // This wrapper is for when you need the nnet
                               // output on CPU, e.g.  because you want it
                               // written to disk.  If this needs to be
                               // configurable in the future, we can make changes
                               // then.
   computer_.SplitUtteranceIntoTasks(
       output_to_cpu, input, ivector, online_ivectors,
       online_ivector_period, &(info->tasks));
 
   // Setting this to a nonzero value will cause the AcceptTask() call below to
   // hang until the computation thread has made some progress, if too much
   // data is already queued.
   int32 max_full_minibatches = 2;
 
   // Earlier utterances have higher priority, which is important to make sure
   // that their corresponding tasks are completed and they can be output to disk.
   double priority = -1.0 * (utterance_counter_++);
   for (size_t i = 0; i < info->tasks.size(); i++) {
     info->tasks[i].priority = priority;
     computer_.AcceptTask(&(info->tasks[i]), max_full_minibatches);
   }
   utts_.push_back(info);
   tasks_ready_semaphore_.Signal();
 }

◆ Compute()

void Compute ( )

private

Definition at line 1171 of file nnet-batch-compute.cc.

References NnetBatchComputer::Compute(), NnetBatchInference::computer_, NnetBatchInference::is_finished_, NnetBatchInference::tasks_ready_semaphore_, and Semaphore::Wait().

                                  {
   bool allow_partial_minibatch = false;
   while (true) {
     // keep calling Compute() as long as it makes progress.
     while (computer_.Compute(allow_partial_minibatch));
 
     // ... then wait on tasks_ready_semaphore_.
     tasks_ready_semaphore_.Wait();
     if (is_finished_) {
       allow_partial_minibatch = true;
       while (computer_.Compute(allow_partial_minibatch));
       return;
     }
   }
 }

◆ ComputeFunc()

static void ComputeFunc ( NnetBatchInference * object )

inlinestaticprivate

Definition at line 565 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::NnetBatchInference().

565 { object->Compute(); }

◆ Finished()

void Finished ( )

The user should call this after the last input has been provided via AcceptInput().

This will force the last utterances to be flushed out (to be retrieved by GetOutput()), rather than waiting until the relevant minibatches are full.

Definition at line 1165 of file nnet-batch-compute.cc.

References NnetBatchInference::is_finished_, Semaphore::Signal(), and NnetBatchInference::tasks_ready_semaphore_.

Referenced by main().

                                   {
   is_finished_ = true;
   tasks_ready_semaphore_.Signal();
 }

◆ GetOutput()

bool GetOutput	(	std::string *	utterance_id,
		Matrix< BaseFloat > *	output
	)

The user should call this to obtain output.

It's guaranteed to be in the same order as the input was provided, but it may be delayed. 'output' will be the output of the neural net, spliced together over the chunks (and with acoustic scaling applied if it was specified in the options; the subtraction of priors will depend whether you supplied a non-empty vector of priors to the constructor.

This call does not block (i.e. does not wait on any semaphores) unless you have previously called Finished(). It returns true if it actually got any output; if none was ready it will return false.

Definition at line 1130 of file nnet-batch-compute.cc.

References NnetBatchInference::is_finished_, kaldi::nnet3::MergeTaskOutput(), NnetBatchInference::UtteranceInfo::num_tasks_finished, NnetBatchInference::UtteranceInfo::tasks, Semaphore::TryWait(), NnetBatchInference::UtteranceInfo::utterance_id, NnetBatchInference::utts_, and Semaphore::Wait().

Referenced by main().

                                                               {
   if (utts_.empty())
     return false;
 
   UtteranceInfo *info = *utts_.begin();
   std::vector<NnetInferenceTask> &tasks = info->tasks;
   int32 num_tasks = tasks.size();
   for (; info->num_tasks_finished < num_tasks; ++info->num_tasks_finished) {
     Semaphore &semaphore = tasks[info->num_tasks_finished].semaphore;
     if (is_finished_) {
       semaphore.Wait();
     } else {
       if (!semaphore.TryWait()) {
         // If not all of the tasks of this utterance are ready yet,
         // just return false.
         return false;
       }
     }
   }
   MergeTaskOutput(tasks, output);
   *utterance_id = info->utterance_id;
   delete info;
   utts_.pop_front();
   return true;
 }

◆ KALDI_DISALLOW_COPY_AND_ASSIGN()

KALDI_DISALLOW_COPY_AND_ASSIGN ( NnetBatchInference )

private

Member Data Documentation

◆ compute_thread_

std::thread compute_thread_

private

Definition at line 602 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::NnetBatchInference(), and NnetBatchInference::~NnetBatchInference().

◆ computer_

NnetBatchComputer computer_

private

Definition at line 572 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::AcceptInput(), and NnetBatchInference::Compute().

◆ is_finished_

bool is_finished_

private

Definition at line 576 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::Compute(), NnetBatchInference::Finished(), NnetBatchInference::GetOutput(), and NnetBatchInference::~NnetBatchInference().

◆ tasks_ready_semaphore_

Semaphore tasks_ready_semaphore_

private

Definition at line 581 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::AcceptInput(), NnetBatchInference::Compute(), and NnetBatchInference::Finished().

◆ utterance_counter_

int32 utterance_counter_

private

Definition at line 599 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::AcceptInput().

◆ utts_

std::list<UtteranceInfo*> utts_

private

Definition at line 597 of file nnet-batch-compute.h.

Referenced by NnetBatchInference::AcceptInput(), NnetBatchInference::GetOutput(), and NnetBatchInference::~NnetBatchInference().

The documentation for this class was generated from the following files:

nnet3/nnet-batch-compute.h
nnet3/nnet-batch-compute.cc

Classes

Public Member Functions

Private Member Functions

Static Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ NnetBatchInference()

◆ ~NnetBatchInference()

Member Function Documentation

◆ AcceptInput()

◆ Compute()

◆ ComputeFunc()

◆ Finished()

◆ GetOutput()

◆ KALDI_DISALLOW_COPY_AND_ASSIGN()

Member Data Documentation

◆ compute_thread_

◆ computer_

◆ is_finished_

◆ tasks_ready_semaphore_

◆ utterance_counter_

◆ utts_