online2-wav-nnet3-latgen-faster.cc File Reference
Include dependency graph for online2-wav-nnet3-latgen-faster.cc:

Go to the source code of this file.

Namespaces

 kaldi
 This code computes Goodness of Pronunciation (GOP) and extracts phone-level pronunciation feature for mispronunciations detection tasks, the reference:
 

Functions

void GetDiagnosticsAndPrintOutput (const std::string &utt, const fst::SymbolTable *word_syms, const CompactLattice &clat, int64 *tot_num_frames, double *tot_like)
 
int main (int argc, char *argv[])
 

Function Documentation

◆ main()

int main ( int  argc,
char *  argv[] 
)

Definition at line 78 of file online2-wav-nnet3-latgen-faster.cc.

References NnetSimpleLoopedComputationOptions::acoustic_scale, fst::AcousticLatticeScale(), OnlineSilenceWeighting::Active(), SingleUtteranceNnet3DecoderTpl< FST >::AdvanceDecoding(), kaldi::nnet3::CollapseModel(), OnlineSilenceWeighting::ComputeCurrentTraceback(), WaveData::Data(), SingleUtteranceNnet3DecoderTpl< FST >::Decoder(), SequentialTableReader< Holder >::Done(), SingleUtteranceNnet3DecoderTpl< FST >::EndpointDetected(), SingleUtteranceNnet3DecoderTpl< FST >::FinalizeDecoding(), NnetSimpleLoopedComputationOptions::frame_subsampling_factor, kaldi::g_num_threads, ParseOptions::GetArg(), OnlineSilenceWeighting::GetDeltaWeights(), kaldi::GetDiagnosticsAndPrintOutput(), SingleUtteranceNnet3DecoderTpl< FST >::GetLattice(), AmNnetSimple::GetNnet(), OnlineNnet2FeaturePipelineInfo::global_cmvn_stats_rxfilename, OnlineIvectorExtractionInfo::greedy_ivector_extractor, RandomAccessTableReader< Holder >::HasKey(), rnnlm::i, OnlineNnet2FeaturePipelineInfo::ivector_extractor_info, KALDI_ERR, KALDI_LOG, KALDI_WARN, SequentialTableReader< Holder >::Key(), SequentialTableReader< Holder >::Next(), ParseOptions::NumArgs(), OnlineTimer::OutputStats(), OnlineTimingStats::Print(), ParseOptions::PrintUsage(), AmNnetSimple::Read(), ParseOptions::Read(), TransitionModel::Read(), fst::ReadFstKaldiGeneric(), kaldi::ReadKaldiObject(), NnetSimpleLoopedComputationOptions::Register(), LatticeFasterDecoderConfig::Register(), ParseOptions::Register(), OnlineNnet2FeaturePipelineConfig::Register(), OnlineEndpointConfig::Register(), WaveData::SampFreq(), fst::ScaleLattice(), kaldi::nnet3::SetBatchnormTestMode(), kaldi::nnet3::SetDropoutTestMode(), OnlineNnet2FeaturePipelineInfo::silence_weighting_config, Input::Stream(), OnlineIvectorExtractionInfo::use_most_recent_ivector, RandomAccessTableReader< Holder >::Value(), SequentialTableReader< Holder >::Value(), OnlineTimer::WaitUntil(), and TableWriter< Holder >::Write().

78  {
79  try {
80  using namespace kaldi;
81  using namespace fst;
82 
83  typedef kaldi::int32 int32;
84  typedef kaldi::int64 int64;
85 
86  const char *usage =
87  "Reads in wav file(s) and simulates online decoding with neural nets\n"
88  "(nnet3 setup), with optional iVector-based speaker adaptation and\n"
89  "optional endpointing. Note: some configuration values and inputs are\n"
90  "set via config files whose filenames are passed as options\n"
91  "\n"
92  "Usage: online2-wav-nnet3-latgen-faster [options] <nnet3-in> <fst-in> "
93  "<spk2utt-rspecifier> <wav-rspecifier> <lattice-wspecifier>\n"
94  "The spk2utt-rspecifier can just be <utterance-id> <utterance-id> if\n"
95  "you want to decode utterance by utterance.\n";
96 
97  ParseOptions po(usage);
98 
99  std::string word_syms_rxfilename;
100 
101  // feature_opts includes configuration for the iVector adaptation,
102  // as well as the basic features.
105  LatticeFasterDecoderConfig decoder_opts;
106  OnlineEndpointConfig endpoint_opts;
107 
108  BaseFloat chunk_length_secs = 0.18;
109  bool do_endpointing = false;
110  bool online = true;
111 
112  po.Register("chunk-length", &chunk_length_secs,
113  "Length of chunk size in seconds, that we process. Set to <= 0 "
114  "to use all input in one chunk.");
115  po.Register("word-symbol-table", &word_syms_rxfilename,
116  "Symbol table for words [for debug output]");
117  po.Register("do-endpointing", &do_endpointing,
118  "If true, apply endpoint detection");
119  po.Register("online", &online,
120  "You can set this to false to disable online iVector estimation "
121  "and have all the data for each utterance used, even at "
122  "utterance start. This is useful where you just want the best "
123  "results and don't care about online operation. Setting this to "
124  "false has the same effect as setting "
125  "--use-most-recent-ivector=true and --greedy-ivector-extractor=true "
126  "in the file given to --ivector-extraction-config, and "
127  "--chunk-length=-1.");
128  po.Register("num-threads-startup", &g_num_threads,
129  "Number of threads used when initializing iVector extractor.");
130 
131  feature_opts.Register(&po);
132  decodable_opts.Register(&po);
133  decoder_opts.Register(&po);
134  endpoint_opts.Register(&po);
135 
136 
137  po.Read(argc, argv);
138 
139  if (po.NumArgs() != 5) {
140  po.PrintUsage();
141  return 1;
142  }
143 
144  std::string nnet3_rxfilename = po.GetArg(1),
145  fst_rxfilename = po.GetArg(2),
146  spk2utt_rspecifier = po.GetArg(3),
147  wav_rspecifier = po.GetArg(4),
148  clat_wspecifier = po.GetArg(5);
149 
150  OnlineNnet2FeaturePipelineInfo feature_info(feature_opts);
151  if (!online) {
152  feature_info.ivector_extractor_info.use_most_recent_ivector = true;
153  feature_info.ivector_extractor_info.greedy_ivector_extractor = true;
154  chunk_length_secs = -1.0;
155  }
156 
157  Matrix<double> global_cmvn_stats;
158  if (feature_info.global_cmvn_stats_rxfilename != "")
159  ReadKaldiObject(feature_info.global_cmvn_stats_rxfilename,
160  &global_cmvn_stats);
161 
162  TransitionModel trans_model;
163  nnet3::AmNnetSimple am_nnet;
164  {
165  bool binary;
166  Input ki(nnet3_rxfilename, &binary);
167  trans_model.Read(ki.Stream(), binary);
168  am_nnet.Read(ki.Stream(), binary);
169  SetBatchnormTestMode(true, &(am_nnet.GetNnet()));
170  SetDropoutTestMode(true, &(am_nnet.GetNnet()));
172  }
173 
174  // this object contains precomputed stuff that is used by all decodable
175  // objects. It takes a pointer to am_nnet because if it has iVectors it has
176  // to modify the nnet to accept iVectors at intervals.
177  nnet3::DecodableNnetSimpleLoopedInfo decodable_info(decodable_opts,
178  &am_nnet);
179 
180 
181  fst::Fst<fst::StdArc> *decode_fst = ReadFstKaldiGeneric(fst_rxfilename);
182 
183  fst::SymbolTable *word_syms = NULL;
184  if (word_syms_rxfilename != "")
185  if (!(word_syms = fst::SymbolTable::ReadText(word_syms_rxfilename)))
186  KALDI_ERR << "Could not read symbol table from file "
187  << word_syms_rxfilename;
188 
189  int32 num_done = 0, num_err = 0;
190  double tot_like = 0.0;
191  int64 num_frames = 0;
192 
193  SequentialTokenVectorReader spk2utt_reader(spk2utt_rspecifier);
194  RandomAccessTableReader<WaveHolder> wav_reader(wav_rspecifier);
195  CompactLatticeWriter clat_writer(clat_wspecifier);
196 
197  OnlineTimingStats timing_stats;
198 
199  for (; !spk2utt_reader.Done(); spk2utt_reader.Next()) {
200  std::string spk = spk2utt_reader.Key();
201  const std::vector<std::string> &uttlist = spk2utt_reader.Value();
202 
203  OnlineIvectorExtractorAdaptationState adaptation_state(
204  feature_info.ivector_extractor_info);
205  OnlineCmvnState cmvn_state(global_cmvn_stats);
206 
207  for (size_t i = 0; i < uttlist.size(); i++) {
208  std::string utt = uttlist[i];
209  if (!wav_reader.HasKey(utt)) {
210  KALDI_WARN << "Did not find audio for utterance " << utt;
211  num_err++;
212  continue;
213  }
214  const WaveData &wave_data = wav_reader.Value(utt);
215  // get the data for channel zero (if the signal is not mono, we only
216  // take the first channel).
217  SubVector<BaseFloat> data(wave_data.Data(), 0);
218 
219  OnlineNnet2FeaturePipeline feature_pipeline(feature_info);
220  feature_pipeline.SetAdaptationState(adaptation_state);
221  feature_pipeline.SetCmvnState(cmvn_state);
222 
223  OnlineSilenceWeighting silence_weighting(
224  trans_model,
225  feature_info.silence_weighting_config,
226  decodable_opts.frame_subsampling_factor);
227 
228  SingleUtteranceNnet3Decoder decoder(decoder_opts, trans_model,
229  decodable_info,
230  *decode_fst, &feature_pipeline);
231  OnlineTimer decoding_timer(utt);
232 
233  BaseFloat samp_freq = wave_data.SampFreq();
234  int32 chunk_length;
235  if (chunk_length_secs > 0) {
236  chunk_length = int32(samp_freq * chunk_length_secs);
237  if (chunk_length == 0) chunk_length = 1;
238  } else {
239  chunk_length = std::numeric_limits<int32>::max();
240  }
241 
242  int32 samp_offset = 0;
243  std::vector<std::pair<int32, BaseFloat> > delta_weights;
244 
245  while (samp_offset < data.Dim()) {
246  int32 samp_remaining = data.Dim() - samp_offset;
247  int32 num_samp = chunk_length < samp_remaining ? chunk_length
248  : samp_remaining;
249 
250  SubVector<BaseFloat> wave_part(data, samp_offset, num_samp);
251  feature_pipeline.AcceptWaveform(samp_freq, wave_part);
252 
253  samp_offset += num_samp;
254  decoding_timer.WaitUntil(samp_offset / samp_freq);
255  if (samp_offset == data.Dim()) {
256  // no more input. flush out last frames
257  feature_pipeline.InputFinished();
258  }
259 
260  if (silence_weighting.Active() &&
261  feature_pipeline.IvectorFeature() != NULL) {
262  silence_weighting.ComputeCurrentTraceback(decoder.Decoder());
263  silence_weighting.GetDeltaWeights(feature_pipeline.NumFramesReady(),
264  &delta_weights);
265  feature_pipeline.IvectorFeature()->UpdateFrameWeights(delta_weights);
266  }
267 
268  decoder.AdvanceDecoding();
269 
270  if (do_endpointing && decoder.EndpointDetected(endpoint_opts)) {
271  break;
272  }
273  }
274  decoder.FinalizeDecoding();
275 
276  CompactLattice clat;
277  bool end_of_utterance = true;
278  decoder.GetLattice(end_of_utterance, &clat);
279 
280  GetDiagnosticsAndPrintOutput(utt, word_syms, clat,
281  &num_frames, &tot_like);
282 
283  decoding_timer.OutputStats(&timing_stats);
284 
285  // In an application you might avoid updating the adaptation state if
286  // you felt the utterance had low confidence. See lat/confidence.h
287  feature_pipeline.GetAdaptationState(&adaptation_state);
288  feature_pipeline.GetCmvnState(&cmvn_state);
289 
290  // we want to output the lattice with un-scaled acoustics.
291  BaseFloat inv_acoustic_scale =
292  1.0 / decodable_opts.acoustic_scale;
293  ScaleLattice(AcousticLatticeScale(inv_acoustic_scale), &clat);
294 
295  clat_writer.Write(utt, clat);
296  KALDI_LOG << "Decoded utterance " << utt;
297  num_done++;
298  }
299  }
300  timing_stats.Print(online);
301 
302  KALDI_LOG << "Decoded " << num_done << " utterances, "
303  << num_err << " with errors.";
304  KALDI_LOG << "Overall likelihood per frame was " << (tot_like / num_frames)
305  << " per frame over " << num_frames << " frames.";
306  delete decode_fst;
307  delete word_syms; // will delete if non-NULL.
308  return (num_done != 0 ? 0 : 1);
309  } catch(const std::exception& e) {
310  std::cerr << e.what();
311  return -1;
312  }
313 } // main()
This code computes Goodness of Pronunciation (GOP) and extracts phone-level pronunciation feature for...
Definition: chain.dox:20
void CollapseModel(const CollapseModelConfig &config, Nnet *nnet)
This function modifies the neural net for efficiency, in a way that suitable to be done in test time...
Definition: nnet-utils.cc:2100
class OnlineTimer is used to test real-time decoding algorithms and evaluate how long the decoding of...
Definition: online-timing.h:88
This configuration class is to set up OnlineNnet2FeaturePipelineInfo, which in turn is the configurat...
Fst< StdArc > * ReadFstKaldiGeneric(std::string rxfilename, bool throw_on_err)
Definition: kaldi-fst-io.cc:45
For an extended explanation of the framework of which grammar-fsts are a part, please see Support for...
Definition: graph.dox:21
int32 g_num_threads
Definition: kaldi-thread.cc:25
This class stores the adaptation state from the online iVector extractor, which can help you to initi...
void SetBatchnormTestMode(bool test_mode, Nnet *nnet)
This function affects only components of type BatchNormComponent.
Definition: nnet-utils.cc:564
A templated class for writing objects to an archive or script file; see The Table concept...
Definition: kaldi-table.h:368
kaldi::int32 int32
BaseFloat SampFreq() const
Definition: wave-reader.h:126
const Matrix< BaseFloat > & Data() const
Definition: wave-reader.h:124
const Nnet & GetNnet() const
void Register(OptionsItf *opts)
void GetDiagnosticsAndPrintOutput(const std::string &utt, const fst::SymbolTable *word_syms, const CompactLattice &clat, int64 *tot_num_frames, double *tot_like)
This class is responsible for storing configuration variables, objects and options for OnlineNnet2Fea...
void Read(std::istream &is, bool binary)
void ReadKaldiObject(const std::string &filename, Matrix< float > *m)
Definition: kaldi-io.cc:832
Allows random access to a collection of objects in an archive or script file; see The Table concept...
Definition: kaldi-table.h:233
void SetDropoutTestMode(bool test_mode, Nnet *nnet)
This function affects components of child-classes of RandomComponent.
Definition: nnet-utils.cc:573
std::vector< std::vector< double > > AcousticLatticeScale(double acwt)
float BaseFloat
Definition: kaldi-types.h:29
The class ParseOptions is for parsing command-line options; see Parsing command-line options for more...
Definition: parse-options.h:36
void Print(bool online=true)
Here, if "online == false" we take into account that the setup was used in not-really-online mode whe...
void ScaleLattice(const std::vector< std::vector< ScaleFloat > > &scale, MutableFst< ArcTpl< Weight > > *fst)
Scales the pairs of weights in LatticeWeight or CompactLatticeWeight by viewing the pair (a...
void Read(std::istream &is, bool binary)
Struct OnlineCmvnState stores the state of CMVN adaptation between utterances (but not the state of t...
A templated class for reading objects sequentially from an archive or script file; see The Table conc...
Definition: kaldi-table.h:287
You will instantiate this class when you want to decode a single utterance using the online-decoding ...
#define KALDI_ERR
Definition: kaldi-error.h:147
#define KALDI_WARN
Definition: kaldi-error.h:150
fst::VectorFst< CompactLatticeArc > CompactLattice
Definition: kaldi-lattice.h:46
This class&#39;s purpose is to read in Wave files.
Definition: wave-reader.h:106
OnlineNnet2FeaturePipeline is a class that&#39;s responsible for putting together the various parts of th...
class OnlineTimingStats stores statistics from timing of online decoding, which will enable the Print...
Definition: online-timing.h:41
#define KALDI_LOG
Definition: kaldi-error.h:153
When you instantiate class DecodableNnetSimpleLooped, you should give it a const reference to this cl...
Represents a non-allocating general vector which can be defined as a sub-vector of higher-level vecto...
Definition: kaldi-vector.h:501
Config class for the CollapseModel function.
Definition: nnet-utils.h:240